scrapeunblocker 0.12.0 → 0.13.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9349609e20a6b8949d1cc3d77b919cf6c916b4b1917d181bb2d18ec441f46777
-  data.tar.gz: 55bf0f292146e6e166a502ecb43a33d15b2629261f808ed8754f4ca7a33d96fe
+  metadata.gz: df0fdd8cb4a59569bb426126e472c5d4139f0ffb8ae38facfed36c6718080975
+  data.tar.gz: 11fc29bb559e468c8a8fdeedd94eb68d3555b73746f9a615859b216fa861ce3c
 SHA512:
-  metadata.gz: 22da248af5842639758605b333930c888534405b796cd199f9c8a2c8bced4104e6dace474c62fbaa2aa878d321878aef3118121fb0563efc4f25a1d2c00451b6
-  data.tar.gz: 3fea75e8eb0602eb156978c019fea92fe7cfca41aab0714f9049412d26207e707899750ad8547b0caf57d3da368d347d2f2ead34e82d74ee2d4764de5f90afb9
+  metadata.gz: 42e64a25aed1a2c912aeff75bae7bacab8d6571bade1549980044744b7e02d855de73e4227b6a7db3c46b95b9c1571257c5423dd8d84c6d95086d22b74d1d4b1
+  data.tar.gz: 2105548d2c9e6fa30492b8e425d786e6b32ebdfe23dd3fff97868422e3a5f6fc2676a5d8115c3e6d438c237eb2238145079b628b914c9229cf162f855023bce9

data/CHANGELOG.md

@@ -1,5 +1,13 @@
 # Changelog
 
+## 0.13.0 (2026-10-23)
+
+- New `get_markdown(url, main_content_only: true, ...)` returns the rendered page as clean Markdown for LLMs, agents and RAG pipelines: headings, lists, tables, absolute links, images with their alt text and code blocks, without scripts, styles, navigation, footers, forms or hidden elements. It sends `markdown=true` to `/getPageSource`, is billed like an HTML call, and returns a UTF-8 `String`.
+- `main_content_only:` (default `true`) keeps only the page's main content, without navigation, site header, footer and sidebars; pass `false` to convert the whole page. The method also takes `proxy_country:`, `time_sleep:`, `method:`, `value:`, `method_timeout:` and `steps:`, like `get_page_source`.
+- A target page that answers 404 or 410 raises `TargetNotFoundError`, as with `get_page_source`; the not-found page, as Markdown, is on `#html`.
+
+No breaking changes.
+
 ## 0.12.0 (2026-10-15)
 
 - `google_images` takes `time_range:` (`"hour"`, `"day"`, `"week"`, `"month"` or `"year"`) to return only the images Google found in that recent period, so a query scanned on a schedule returns only what is new. For a fixed window pass `date_from:` and/or `date_to:` (`"YYYY-MM-DD"`) instead; the API answers 400 when `time_range` is combined with either.

data/README.md

@@ -62,6 +62,21 @@ html = su.get_page_source(
 )
 ```
 
+## Markdown for LLMs
+
+Get the rendered page as clean Markdown instead of HTML, ready for an LLM prompt, an agent or a RAG pipeline. Headings, lists, tables, absolute links and images with their alt text are kept; scripts, styles, navigation, footers and forms are dropped. Billed like an HTML call.
+
+```ruby
+md = su.get_markdown("https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html")
+puts md # "# A Light in the Attic\n\n£51.77\n..." (a UTF-8 String)
+
+# Only the main content by default (no navigation, site header, footer or sidebars).
+# Pass main_content_only: false to convert the whole page:
+full = su.get_markdown("https://example.com", main_content_only: false)
+```
+
+`get_markdown` takes the same `proxy_country:`, `time_sleep:`, `method:` / `value:` / `method_timeout:` and `steps:` options as `get_page_source`. More in the [Markdown guide](https://docs.scrapeunblocker.com/guides/markdown?utm_source=rubygems&utm_medium=integration&utm_campaign=ruby-sdk).
+
 ## Browser steps
 
 Drive the page after it loads with an ordered list of browser actions - fill a form, click through, wait for content, then capture the rendered result. Steps run once in the real browser, in order.
@@ -299,7 +314,7 @@ end
 | `NotFoundError` | 404 | What you asked for does not exist - a plugin lookup found nothing |
 | `TargetNotFoundError` | 404 / 410 | The target page itself does not exist; carries `origin_status`, `html`, `destination_url` (subclass of `NotFoundError`, billed) |
 | `BrowserTimeoutError` | 408 | Our browser run timed out before the page was ready |
-| `UnsupportedContentError` | 415 | The URL serves something other than HTML (`get_page_source`) or other than an image (`get_image`) |
+| `UnsupportedContentError` | 415 | The URL serves something other than HTML (`get_page_source`, `get_markdown`) or other than an image (`get_image`) |
 | `ValidationError` | 422 | Missing or wrong-typed parameter; `body` holds the `detail` array |
 | `RateLimitError` | 429 | Too many requests |
 | `UpstreamOutageError` | 503 | The target origin is down |
@@ -322,7 +337,7 @@ end
 
 `TargetNotFoundError` subclasses `NotFoundError`, so `rescue ScrapeUnblocker::NotFoundError` catches it too. A 404 without `X-Origin-Status` is the API's own and stays a plain `NotFoundError`.
 
-With `get_parsed` the body is the parsed-data JSON (`{"data": {"page_type": "not_found", ...}}`), so `html` is `nil` there; the raw JSON is on `#body`.
+With `get_parsed` the body is the parsed-data JSON (`{"data": {"page_type": "not_found", ...}}`), so `html` is `nil` there; the raw JSON is on `#body`. With `get_markdown` the not-found page on `#html` is Markdown.
 
 ### No structured data on the page
 

data/lib/scrapeunblocker/client.rb

@@ -75,6 +75,35 @@ module ScrapeUnblocker
       body
     end
 
+    # Fetch a URL and return the rendered page as clean Markdown (a UTF-8 String).
+    #
+    # Built for LLMs, agents and RAG pipelines. The page is rendered in a real
+    # browser behind the same anti-bot bypass chain as #get_page_source, then
+    # converted to Markdown: headings, lists, tables, absolute links, images with
+    # their alt text and code blocks, without scripts, styles, navigation,
+    # footers, forms or hidden elements. Billed like an HTML call.
+    #
+    # +main_content_only+ (default true) keeps only the page's main content,
+    # without navigation, the site header, footer and sidebars; pass false to
+    # convert the whole page. +proxy_country+, +time_sleep+, +method+, +value+,
+    # +method_timeout+ and +steps+ work as in #get_page_source.
+    #
+    # When the target page itself answers 404 or 410 this raises
+    # ScrapeUnblocker::TargetNotFoundError (billed; the not-found page, as
+    # Markdown, is on #html).
+    def get_markdown(url, main_content_only: true, proxy_country: nil, time_sleep: nil, method: nil, value: nil,
+                     method_timeout: nil, steps: nil)
+      body = request("/getPageSource",
+                     url: url, markdown: true,
+                     main_content_only: (main_content_only == false ? false : nil),
+                     proxy_country: proxy_country, time_sleep: time_sleep,
+                     method: method, value: value, method_timeout: method_timeout,
+                     steps: (steps ? JSON.generate(steps) : nil))[:body]
+      # The API always sends Markdown as UTF-8; Net::HTTP hands the body over
+      # as binary, which breaks string operations on non-ASCII text.
+      body.dup.force_encoding(Encoding::UTF_8)
+    end
+
     # Fetch a URL and return structured JSON instead of HTML.
     #
     # A page that rendered but held no structured data comes back as a

data/lib/scrapeunblocker/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ScrapeUnblocker
-  VERSION = "0.12.0"
+  VERSION = "0.13.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: scrapeunblocker
 version: !ruby/object:Gem::Version
-  version: 0.12.0
+  version: 0.13.0
 platform: ruby
 authors:
 - ScrapeUnblocker
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-10-15 00:00:00.000000000 Z
+date: 2026-10-23 00:00:00.000000000 Z
 dependencies: []
 description: JS-rendered pages that bypass Cloudflare, DataDome, PerimeterX and Akamai,
   plus Google SERP and 100+ ready-made site plugins (Amazon, Walmart, Zillow, Google