scrapeunblocker 0.11.1 → 0.13.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9bd6d663c941e4d49cc8c8b7446f15c2c4cff0ac7025e85144dd72706ac24220
-  data.tar.gz: 68949c187f57eb403d1f890a236fd9caa671beeaf395225c8100d4da22ce8fac
+  metadata.gz: df0fdd8cb4a59569bb426126e472c5d4139f0ffb8ae38facfed36c6718080975
+  data.tar.gz: 11fc29bb559e468c8a8fdeedd94eb68d3555b73746f9a615859b216fa861ce3c
 SHA512:
-  metadata.gz: b6c58655e6ea4657b358e0edf1c147226d2fdbb5b55b64a18b88f3f5b5d821b58b78ef29ccb2f505554998c6ef9e8183afa45f5dcf01fcf9f9eb3fb5dea28d75
-  data.tar.gz: 8ab2b5635e0b7edc16bfd1202e92ed24e0a189a115d2f37ea26e2c21c9b7780e5f9f942a50edda29cdb6645e2b7a61228bcea014abe08c44f20f5565f31f1057
+  metadata.gz: 42e64a25aed1a2c912aeff75bae7bacab8d6571bade1549980044744b7e02d855de73e4227b6a7db3c46b95b9c1571257c5423dd8d84c6d95086d22b74d1d4b1
+  data.tar.gz: 2105548d2c9e6fa30492b8e425d786e6b32ebdfe23dd3fff97868422e3a5f6fc2676a5d8115c3e6d438c237eb2238145079b628b914c9229cf162f855023bce9

data/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## 0.13.0 (2026-10-23)
+
+- New `get_markdown(url, main_content_only: true, ...)` returns the rendered page as clean Markdown for LLMs, agents and RAG pipelines: headings, lists, tables, absolute links, images with their alt text and code blocks, without scripts, styles, navigation, footers, forms or hidden elements. It sends `markdown=true` to `/getPageSource`, is billed like an HTML call, and returns a UTF-8 `String`.
+- `main_content_only:` (default `true`) keeps only the page's main content, without navigation, site header, footer and sidebars; pass `false` to convert the whole page. The method also takes `proxy_country:`, `time_sleep:`, `method:`, `value:`, `method_timeout:` and `steps:`, like `get_page_source`.
+- A target page that answers 404 or 410 raises `TargetNotFoundError`, as with `get_page_source`; the not-found page, as Markdown, is on `#html`.
+
+No breaking changes.
+
+## 0.12.0 (2026-10-15)
+
+- `google_images` takes `time_range:` (`"hour"`, `"day"`, `"week"`, `"month"` or `"year"`) to return only the images Google found in that recent period, so a query scanned on a schedule returns only what is new. For a fixed window pass `date_from:` and/or `date_to:` (`"YYYY-MM-DD"`) instead; the API answers 400 when `time_range` is combined with either.
+
+No breaking changes.
+
 ## 0.11.1 (2026-10-15)
 
 - `get_image` docs: the API now returns the image's original bytes and format (JPEG, PNG, WebP, AVIF, GIF, SVG, ...) instead of a PNG re-encode. The method still returns a binary `String`.

data/README.md

@@ -62,6 +62,21 @@ html = su.get_page_source(
 )
 ```
 
+## Markdown for LLMs
+
+Get the rendered page as clean Markdown instead of HTML, ready for an LLM prompt, an agent or a RAG pipeline. Headings, lists, tables, absolute links and images with their alt text are kept; scripts, styles, navigation, footers and forms are dropped. Billed like an HTML call.
+
+```ruby
+md = su.get_markdown("https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html")
+puts md # "# A Light in the Attic\n\n£51.77\n..." (a UTF-8 String)
+
+# Only the main content by default (no navigation, site header, footer or sidebars).
+# Pass main_content_only: false to convert the whole page:
+full = su.get_markdown("https://example.com", main_content_only: false)
+```
+
+`get_markdown` takes the same `proxy_country:`, `time_sleep:`, `method:` / `value:` / `method_timeout:` and `steps:` options as `get_page_source`. More in the [Markdown guide](https://docs.scrapeunblocker.com/guides/markdown?utm_source=rubygems&utm_medium=integration&utm_campaign=ruby-sdk).
+
 ## Browser steps
 
 Drive the page after it loads with an ordered list of browser actions - fill a form, click through, wait for content, then capture the rendered result. Steps run once in the real browser, in order.
@@ -174,8 +189,15 @@ local["results"].each { |biz| puts "#{biz['name']} #{biz['rating']} #{biz['addre
 ```ruby
 images = su.google_images("golden retriever puppy", proxy_country: "US", pages: 3)
 images["results"].each { |img| puts "#{img['imageUrl']} #{img['sourceDomain']} #{img['title']}" }
+
+# Only images Google found in the last day (for a daily scan of the same query)
+new_images = su.google_images("eiffel tower", proxy_country: "US", time_range: "day")
+# Or a fixed window: date_from / date_to (YYYY-MM-DD), each one optional
+october = su.google_images("eiffel tower", date_from: "2026-10-01", date_to: "2026-10-07")
 ```
 
+`time_range` takes `hour`, `day`, `week`, `month` or `year` and cannot be combined with `date_from` / `date_to`.
+
 ## Meta Ad Library
 
 ```ruby
@@ -292,7 +314,7 @@ end
 | `NotFoundError` | 404 | What you asked for does not exist - a plugin lookup found nothing |
 | `TargetNotFoundError` | 404 / 410 | The target page itself does not exist; carries `origin_status`, `html`, `destination_url` (subclass of `NotFoundError`, billed) |
 | `BrowserTimeoutError` | 408 | Our browser run timed out before the page was ready |
-| `UnsupportedContentError` | 415 | The URL serves something other than HTML (`get_page_source`) or other than an image (`get_image`) |
+| `UnsupportedContentError` | 415 | The URL serves something other than HTML (`get_page_source`, `get_markdown`) or other than an image (`get_image`) |
 | `ValidationError` | 422 | Missing or wrong-typed parameter; `body` holds the `detail` array |
 | `RateLimitError` | 429 | Too many requests |
 | `UpstreamOutageError` | 503 | The target origin is down |
@@ -315,7 +337,7 @@ end
 
 `TargetNotFoundError` subclasses `NotFoundError`, so `rescue ScrapeUnblocker::NotFoundError` catches it too. A 404 without `X-Origin-Status` is the API's own and stays a plain `NotFoundError`.
 
-With `get_parsed` the body is the parsed-data JSON (`{"data": {"page_type": "not_found", ...}}`), so `html` is `nil` there; the raw JSON is on `#body`.
+With `get_parsed` the body is the parsed-data JSON (`{"data": {"page_type": "not_found", ...}}`), so `html` is `nil` there; the raw JSON is on `#body`. With `get_markdown` the not-found page on `#html` is Markdown.
 
 ### No structured data on the page
 

data/lib/scrapeunblocker/client.rb

@@ -75,6 +75,35 @@ module ScrapeUnblocker
       body
     end
 
+    # Fetch a URL and return the rendered page as clean Markdown (a UTF-8 String).
+    #
+    # Built for LLMs, agents and RAG pipelines. The page is rendered in a real
+    # browser behind the same anti-bot bypass chain as #get_page_source, then
+    # converted to Markdown: headings, lists, tables, absolute links, images with
+    # their alt text and code blocks, without scripts, styles, navigation,
+    # footers, forms or hidden elements. Billed like an HTML call.
+    #
+    # +main_content_only+ (default true) keeps only the page's main content,
+    # without navigation, the site header, footer and sidebars; pass false to
+    # convert the whole page. +proxy_country+, +time_sleep+, +method+, +value+,
+    # +method_timeout+ and +steps+ work as in #get_page_source.
+    #
+    # When the target page itself answers 404 or 410 this raises
+    # ScrapeUnblocker::TargetNotFoundError (billed; the not-found page, as
+    # Markdown, is on #html).
+    def get_markdown(url, main_content_only: true, proxy_country: nil, time_sleep: nil, method: nil, value: nil,
+                     method_timeout: nil, steps: nil)
+      body = request("/getPageSource",
+                     url: url, markdown: true,
+                     main_content_only: (main_content_only == false ? false : nil),
+                     proxy_country: proxy_country, time_sleep: time_sleep,
+                     method: method, value: value, method_timeout: method_timeout,
+                     steps: (steps ? JSON.generate(steps) : nil))[:body]
+      # The API always sends Markdown as UTF-8; Net::HTTP hands the body over
+      # as binary, which breaks string operations on non-ASCII text.
+      body.dup.force_encoding(Encoding::UTF_8)
+    end
+
     # Fetch a URL and return structured JSON instead of HTML.
     #
     # A page that rendered but held no structured data comes back as a
@@ -126,10 +155,16 @@ module ScrapeUnblocker
     # +proxy_country+ to target a market (the Google market follows it; +gl+
     # is an optional override), +pages+ (1-5, ~100 results each; each page
     # fetched is billed as one request, see +pagesFetched+) and +max_results+
-    # (1-500) to cap the count.
-    def google_images(q, proxy_country: nil, pages: nil, gl: nil, max_results: nil)
+    # (1-500) to cap the count. +time_range+ ("hour", "day", "week", "month",
+    # "year") keeps only the images Google found in that recent period, e.g.
+    # the new ones since the last scheduled scan; +date_from+ / +date_to+
+    # ("YYYY-MM-DD", each optional) set a fixed window instead. The two forms
+    # cannot be combined.
+    def google_images(q, proxy_country: nil, pages: nil, gl: nil, max_results: nil,
+                      time_range: nil, date_from: nil, date_to: nil)
       post_json("/images/google-search",
-                q: q, pages: pages, gl: gl, max_results: max_results, proxy_country: proxy_country)
+                q: q, pages: pages, gl: gl, max_results: max_results, proxy_country: proxy_country,
+                time_range: time_range, date_from: date_from, date_to: date_to)
     end
 
     # Fetch an advertiser's Meta (Facebook) Ad Library ads and return them as a Hash.

data/lib/scrapeunblocker/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ScrapeUnblocker
-  VERSION = "0.11.1"
+  VERSION = "0.13.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: scrapeunblocker
 version: !ruby/object:Gem::Version
-  version: 0.11.1
+  version: 0.13.0
 platform: ruby
 authors:
 - ScrapeUnblocker
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-10-15 00:00:00.000000000 Z
+date: 2026-10-23 00:00:00.000000000 Z
 dependencies: []
 description: JS-rendered pages that bypass Cloudflare, DataDome, PerimeterX and Akamai,
   plus Google SERP and 100+ ready-made site plugins (Amazon, Walmart, Zillow, Google