scrapeunblocker 0.1.9 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 28108195aac1cc524421665dfd8eafcf8c41e0419f0e692172b72f55ab3912c8
-  data.tar.gz: 9f80e41ef6047c3b12e32338739c9edd3a0d796211933a09f47b5dd3b70708e8
+  metadata.gz: d921e3453928431b4e1715efd3f8ae53fa1175f5b4fdbf4a67db12c0409cab18
+  data.tar.gz: 59bcc76323226693b74aa96d8a3f969118f6df95328a6e3424bed1ec8dfa78e0
 SHA512:
-  metadata.gz: 04040b3a41778d5a98307082d147ca90be55992453dd64409e09b94a7cbee9b5c767585fccb3150d426505ba3f6f533215dafdf63a2e675ce024f6909fe2d925
-  data.tar.gz: 65a01b202aea9d3e94126692df4de80d35312821ee6a14425c6cca8f63e2cf59a0d8f51c38cd1e54e972cbe844e80e0b1a461b38948ce8855682d5ad6a7d67c3
+  metadata.gz: 4d0791186d046ebe36897a987c7fd08fdfa98c1c78754fc7a32bc9552ac782a903ec1e10bde22fa7d1b442972586ab8a85d6777a00dc5d310e1c9cfe40f9e372
+  data.tar.gz: b132825a15081702a1e76ab431ae9835f88ffa7a589241775688f1820ee58c14d438e0a9150afa995e70fee3ee97df63de6f0dd6b9ba78c17adbacd0ba00510b

data/CHANGELOG.md

@@ -1,5 +1,12 @@
 # Changelog
 
+## 0.2.0 (2026-08-29)
+
+- `get_page_source` now accepts `steps:` - an ordered Array of browser-action Hashes the API runs in the real browser after the page loads (`wait_for`, `wait_for_text`, `wait`, `click`, `type`, `select`, `press_key`, `scroll`). The array is JSON-encoded into the `steps` query parameter. Steps run once and are not idempotent; a failed step returns HTTP 422 and raises `ScrapeUnblocker::ValidationError`, whose `body` names the failed step (`step_index`, `action`, `reason`, `selector`, `html`).
+- `get_page_source` now accepts `list_elements:` - pass `true` to get a JSON summary of the matched elements (`{"url", "count", "elements"}`) instead of HTML. When set, the method returns the parsed Hash rather than an HTML String, mirroring `get_parsed`.
+
+No breaking changes.
+
 ## 0.1.9 (2026-08-28)
 
 - Added `amazon_product` and `amazon_search` for the new Amazon plugin. `amazon_product(asin:/url:)` returns one product - title, brand, numeric price and currency, list price and savings, availability, rating, review count, seller, feature bullets, categories and images. `amazon_search(keyword, ...)` returns a keyword search's cards - asin, title, price, list price, rating, review count, a clean product URL, image and the sponsored/prime flags - on any of 20 regional marketplaces.

data/README.md

@@ -62,6 +62,47 @@ html = su.get_page_source(
 )
 ```
 
+## Browser steps
+
+Drive the page after it loads with an ordered list of browser actions - fill a form, click through, wait for content, then capture the rendered result. Steps run once in the real browser, in order.
+
+```ruby
+html = su.get_page_source(
+  "https://example.com/search",
+  steps: [
+    { action: "type", selector: "input#q", value: "web scraping", clear: true },
+    { action: "press_key", value: "Enter" },
+    { action: "wait_for", selector: "#results" },
+    { action: "scroll", value: "bottom" }
+  ]
+)
+```
+
+Supported actions and their fields:
+
+| Action | Fields |
+|---|---|
+| `wait_for` | `selector`, `selector_type` (`css` default, `xPath`, `className`, `tagName`), `timeout_ms` |
+| `wait_for_text` | `value` (text to await), `timeout_ms` |
+| `wait` | `value` (milliseconds) |
+| `click` | `selector`, `selector_type`, `timeout_ms` |
+| `type` | `selector`, `selector_type`, `value`, `clear` (bool), `timeout_ms` - types like a human |
+| `select` | `selector`, `selector_type`, `value`, `timeout_ms` |
+| `press_key` | `value` - one of `Enter`, `Tab`, `Escape`, `Backspace`, `Delete`, `Space`, `ArrowUp`, `ArrowDown`, `ArrowLeft`, `ArrowRight`, `Home`, `End`, `PageUp`, `PageDown` |
+| `scroll` | `value` - `"bottom"` or a pixel count |
+
+Steps are not idempotent (they submit forms, click buttons), so a call is not safe to blindly retry. If a step fails, the API answers HTTP 422 and the client raises `ScrapeUnblocker::ValidationError`; its `body` is the JSON describing which step failed (`error: "step_failed"`, `step_index`, `action`, `reason`, `selector`, `html`).
+
+## List elements
+
+Pass `list_elements: true` to get a JSON summary of the matched elements instead of the full HTML. The method then returns a parsed Hash (`{"url", "count", "elements"}`) rather than an HTML String.
+
+```ruby
+result = su.get_page_source("https://example.com", list_elements: true)
+puts result["count"]
+result["elements"].each { |el| p el }
+```
+
 ## Get parsed JSON
 
 ```ruby

data/lib/scrapeunblocker/client.rb

@@ -37,10 +37,31 @@ module ScrapeUnblocker
     end
 
     # Fetch a URL and return the fully rendered HTML.
-    def get_page_source(url, proxy_country: nil, time_sleep: nil, method: nil, value: nil, method_timeout: nil)
-      request("/getPageSource",
-              url: url, proxy_country: proxy_country, time_sleep: time_sleep,
-              method: method, value: value, method_timeout: method_timeout)[:body]
+    #
+    # +steps+ is an ordered Array of browser-action Hashes the API runs in the
+    # real browser after the page loads (each Hash carries an +action+ and its
+    # fields): +wait_for+ {selector, selector_type?, timeout_ms?}, +wait_for_text+
+    # {value, timeout_ms?}, +wait+ {value}, +click+ {selector, selector_type?,
+    # timeout_ms?}, +type+ {selector, selector_type?, value, clear?, timeout_ms?},
+    # +select+ {selector, selector_type?, value, timeout_ms?}, +press_key+ {value},
+    # +scroll+ {value}. +selector_type+ is one of "css" (default), "xPath",
+    # "className" or "tagName". The steps run once and are not idempotent; if a
+    # step fails the API answers HTTP 422 with a JSON body naming the failed step,
+    # which surfaces here as a ScrapeUnblocker::ValidationError.
+    #
+    # +list_elements+, when true, makes the API return a JSON summary of the
+    # matched elements ({"url", "count", "elements"}) instead of HTML. This method
+    # then returns that parsed Hash rather than an HTML String.
+    def get_page_source(url, proxy_country: nil, time_sleep: nil, method: nil, value: nil, method_timeout: nil,
+                        steps: nil, list_elements: nil)
+      body = request("/getPageSource",
+                     url: url, proxy_country: proxy_country, time_sleep: time_sleep,
+                     method: method, value: value, method_timeout: method_timeout,
+                     steps: (steps ? JSON.generate(steps) : nil),
+                     list_elements: (list_elements ? true : nil))[:body]
+      return JSON.parse(body) if list_elements
+
+      body
     end
 
     # Fetch a URL and return structured JSON instead of HTML.

data/lib/scrapeunblocker/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ScrapeUnblocker
-  VERSION = "0.1.9"
+  VERSION = "0.2.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: scrapeunblocker
 version: !ruby/object:Gem::Version
-  version: 0.1.9
+  version: 0.2.0
 platform: ruby
 authors:
 - ScrapeUnblocker
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-08-28 00:00:00.000000000 Z
+date: 2026-08-29 00:00:00.000000000 Z
 dependencies: []
 description: JS-rendered pages that bypass Cloudflare, DataDome, PerimeterX and Akamai,
   plus Google SERP and Skyscanner flights/hotels/car-hire scraping as JSON.