scrapeunblocker 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 9c7871e5c3c3e31003c4a603ec9fda0784a79becb2b2617c922b89a53c39dc43
+  data.tar.gz: f784ea8c8411b97e6dd136aff9e39024d16a514fa059c516951d87e6d30e9874
+SHA512:
+  metadata.gz: 8ce9fb00f43d3ab2f8433f880449581bf477d1bf90ee83036dc0384612cfbe385c6f112fc26407917dbc386629f3ed2e255cca68190dee832b637760e579c738
+  data.tar.gz: c56409262b4d73ff538bd39bb54f7cf2f417c80ba974a467fcafe91fc647b2bdce2ddf1d41b722ef09e949b73c3548f0834487ca09517832b9493946bfbf5a45

data/CHANGELOG.md

@@ -0,0 +1,9 @@
+# Changelog
+
+## 0.1.0
+
+Initial release.
+
+- `ScrapeUnblocker::Client` with `get_page_source`, `get_parsed`, `get_page_with_cookies`, `serp`, `get_image`.
+- Skyscanner plugins: flights, hotels, car hire (quotes + locations).
+- Typed error hierarchy and automatic retry on transient failures.

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 ScrapeUnblocker
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,161 @@
+# ScrapeUnblocker Ruby client
+
+Official Ruby client for the [ScrapeUnblocker](https://scrapeunblocker.com) web scraping API.
+
+Every request is fully JavaScript-rendered in a real browser and routed through premium proxies, so it bypasses Cloudflare, DataDome, PerimeterX, Akamai, Kasada and similar anti-bot systems - from one simple call. You are only billed for successful requests.
+
+- **Highest success rate on the market** (95%+ on live production traffic)
+- **Rendered HTML or parsed JSON** - no per-site parsers to maintain
+- Zero dependencies (uses the standard library), typed errors
+
+## Install
+
+```bash
+gem install scrapeunblocker
+```
+
+Or in a Gemfile:
+
+```ruby
+gem "scrapeunblocker"
+```
+
+Requires Ruby 2.7+.
+
+## Quickstart
+
+```ruby
+require "scrapeunblocker"
+
+su = ScrapeUnblocker::Client.new # reads SCRAPEUNBLOCKER_KEY, or Client.new(api_key: "YOUR_API_KEY")
+
+# Rendered HTML for any URL
+html = su.get_page_source("https://example.com")
+
+# Structured JSON instead of HTML (products, listings, search results, ...)
+product = su.get_parsed("https://www.amazon.com/dp/B08N5WRWNW")
+puts product.page_type # "product"
+p product.data
+```
+
+Get your API key at [app.scrapeunblocker.com](https://app.scrapeunblocker.com). The free trial does not require a credit card.
+
+## Authentication
+
+Set an environment variable and the client picks it up:
+
+```bash
+export SCRAPEUNBLOCKER_KEY="YOUR_API_KEY"
+```
+
+```ruby
+su = ScrapeUnblocker::Client.new # reads SCRAPEUNBLOCKER_KEY
+```
+
+## Fetch rendered HTML
+
+```ruby
+html = su.get_page_source(
+  "https://www.nordstrom.com/browse/women/clothing/dresses",
+  proxy_country: "US", # route through a specific country
+  time_sleep: 3        # wait extra seconds after load
+)
+```
+
+## Get parsed JSON
+
+```ruby
+result = su.get_parsed("https://www.walmart.com/ip/12345")
+puts result.page_type # e.g. "product"
+puts result.source    # how it was extracted
+p result.data         # the fields
+
+# If a parse ever comes back wrong, force a fresh set of rules:
+fresh = su.get_parsed(url, refresh_rules: true, rules_hint: "price is missing")
+```
+
+## Google search (SERP)
+
+```ruby
+serp = su.serp("web scraping api", pages_to_check: 2, proxy_country: "US")
+```
+
+## Cookies and the serving proxy
+
+```ruby
+page = su.get_page_with_cookies("https://example.com")
+puts page.html
+p page.cookies
+puts page.proxy
+```
+
+## Images
+
+```ruby
+bytes = su.get_image("https://example.com/photo.jpg")
+File.binwrite("photo.jpg", bytes)
+```
+
+## Skyscanner plugins
+
+Flights, hotels and car hire as JSON:
+
+```ruby
+locations = su.skyscanner.flight_locations("London")
+
+flights = su.skyscanner.flights(
+  origin: "London", dest: "New York",
+  depart_date: "2026-09-01", adults: 1, currency: "USD"
+)
+
+hotels = su.skyscanner.hotels(destination: "Madrid", checkin: "2026-09-01", checkout: "2026-09-03")
+cars = su.skyscanner.carhire(pickup: "Madrid", pickup_datetime: "2026-09-01T10:00", dropoff_datetime: "2026-09-03T10:00")
+```
+
+## Error handling
+
+Non-2xx responses raise typed errors, all subclasses of `ScrapeUnblocker::Error`. Transient failures (429, 502, 503, 504 and network errors) are retried automatically with exponential backoff.
+
+```ruby
+begin
+  html = su.get_page_source("https://example.com")
+rescue ScrapeUnblocker::BlockedError
+  # 403: the target blocked every bypass path (not billed)
+rescue ScrapeUnblocker::RateLimitError
+  # 429: slow down
+rescue ScrapeUnblocker::UpstreamOutageError
+  # 503: the target site itself is down - retry later
+end
+```
+
+| Error | Status | Meaning |
+|---|---|---|
+| `InvalidRequestError` | 400 | Bad URL or unsupported scheme |
+| `AuthenticationError` | 401 | Missing or invalid API key |
+| `BlockedError` | 403 | Blocked by bot protection on every path |
+| `RateLimitError` | 429 | Too many requests |
+| `UpstreamOutageError` | 503 | The target origin is down |
+| `ServerError` | 5xx | Unexpected server error |
+| `TimeoutError` | - | Request exceeded the timeout |
+| `ConnectionError` | - | Could not reach the API |
+
+## Configuration
+
+```ruby
+ScrapeUnblocker::Client.new(
+  api_key: nil,        # or SCRAPEUNBLOCKER_KEY env var
+  base_url: "https://api.scrapeunblocker.com",
+  timeout: 180,        # seconds; protected pages can be slow
+  max_retries: 2
+)
+```
+
+## Links
+
+- Documentation: https://developers.scrapeunblocker.com
+- Website: https://scrapeunblocker.com
+- Dashboard: https://app.scrapeunblocker.com
+
+## License
+
+MIT

data/lib/scrapeunblocker/client.rb

@@ -0,0 +1,141 @@
+# frozen_string_literal: true
+
+require "net/http"
+require "uri"
+require "json"
+
+require_relative "errors"
+require_relative "parsed_page"
+require_relative "skyscanner"
+require_relative "version"
+
+module ScrapeUnblocker
+  # Client for the ScrapeUnblocker API.
+  #
+  #   su = ScrapeUnblocker::Client.new(api_key: "YOUR_API_KEY")
+  #   html = su.get_page_source("https://example.com")
+  class Client
+    DEFAULT_BASE_URL = "https://api.scrapeunblocker.com"
+    API_KEY_HEADER = "x-scrapeunblocker-key"
+    RETRYABLE = [429, 502, 503, 504].freeze
+
+    # @return [Skyscanner] the Skyscanner plugin endpoints
+    attr_reader :skyscanner
+
+    def initialize(api_key: nil, base_url: DEFAULT_BASE_URL, timeout: 180, max_retries: 2, transport: nil)
+      @api_key = api_key || ENV["SCRAPEUNBLOCKER_KEY"]
+      if @api_key.nil? || @api_key.empty?
+        raise Error, "No API key provided. Pass api_key: or set the " \
+                     "SCRAPEUNBLOCKER_KEY environment variable. Get your key " \
+                     "at https://app.scrapeunblocker.com"
+      end
+      @base_url = base_url.sub(%r{/+\z}, "")
+      @timeout = timeout
+      @max_retries = max_retries
+      @transport = transport || method(:net_http_transport)
+      @skyscanner = Skyscanner.new(self)
+    end
+
+    # Fetch a URL and return the fully rendered HTML.
+    def get_page_source(url, proxy_country: nil, time_sleep: nil, method: nil, value: nil, method_timeout: nil)
+      request("/getPageSource",
+              url: url, proxy_country: proxy_country, time_sleep: time_sleep,
+              method: method, value: value, method_timeout: method_timeout)[:body]
+    end
+
+    # Fetch a URL and return structured JSON instead of HTML.
+    def get_parsed(url, proxy_country: nil, time_sleep: nil, refresh_rules: false, rules_hint: nil)
+      body = request("/getPageSource",
+                     url: url, parsed_data: true, proxy_country: proxy_country,
+                     time_sleep: time_sleep,
+                     refresh_rules: (refresh_rules ? true : nil),
+                     rules_hint: rules_hint)[:body]
+      ParsedPage.from_hash(JSON.parse(body))
+    end
+
+    # Fetch a URL and also return the cookies and proxy that served it.
+    def get_page_with_cookies(url, proxy_country: nil, time_sleep: nil)
+      body = request("/getPageSource",
+                     url: url, get_cookies: true, proxy_country: proxy_country,
+                     time_sleep: time_sleep)[:body]
+      PageResult.from_hash(JSON.parse(body))
+    end
+
+    # Run a Google search and return the parsed SERP as a Hash.
+    def serp(keyword, proxy_country: nil, pages_to_check: 1, wait_after_load: 0, captcha_pause: 0)
+      post_json("/serpApi",
+                keyword: keyword, proxy_country: proxy_country,
+                pages_to_check: pages_to_check,
+                wait_after_load: (wait_after_load.zero? ? nil : wait_after_load),
+                captcha_pause: (captcha_pause.zero? ? nil : captcha_pause))
+    end
+
+    # Fetch an image URL through the bypass chain and return its raw bytes.
+    def get_image(url, proxy_country: nil)
+      request("/getImage", url: url, proxy_country: proxy_country)[:body]
+    end
+
+    # @api private
+    def post_json(path, params)
+      JSON.parse(request(path, params)[:body])
+    end
+
+    private
+
+    def request(path, params)
+      url = "#{@base_url}#{path}?#{build_query(params)}"
+      headers = {
+        API_KEY_HEADER => @api_key,
+        "User-Agent" => "scrapeunblocker-ruby/#{VERSION}",
+        "Accept" => "*/*"
+      }
+
+      attempt = 0
+      loop do
+        result = @transport.call(url, headers)
+        status = result[:status].to_i
+        body = result[:body].to_s
+
+        if RETRYABLE.include?(status) && attempt < @max_retries
+          sleep([0.5 * (2**attempt), 8.0].min)
+          attempt += 1
+          next
+        end
+
+        return { status: status, body: body } if status >= 200 && status < 300
+
+        raise ScrapeUnblocker.error_for_status(status, body)
+      end
+    end
+
+    def build_query(params)
+      params.each_with_object([]) do |(key, value), acc|
+        next if value.nil?
+
+        value = value ? "true" : "false" if value == true || value == false
+        acc << "#{key}=#{URI.encode_www_form_component(value.to_s)}"
+      end.join("&")
+    end
+
+    def net_http_transport(url, headers)
+      uri = URI.parse(url)
+      http = Net::HTTP.new(uri.host, uri.port)
+      http.use_ssl = uri.scheme == "https"
+      http.open_timeout = 30
+      http.read_timeout = @timeout
+
+      request = Net::HTTP::Post.new(uri.request_uri)
+      headers.each { |k, v| request[k] = v }
+
+      begin
+        response = http.request(request)
+      rescue Net::OpenTimeout, Net::ReadTimeout => e
+        raise TimeoutError, "Request timed out after #{@timeout}s: #{e.message}"
+      rescue StandardError => e
+        raise ConnectionError, "Could not reach the API: #{e.message}"
+      end
+
+      { status: response.code.to_i, body: response.body }
+    end
+  end
+end

data/lib/scrapeunblocker/errors.rb

@@ -0,0 +1,68 @@
+# frozen_string_literal: true
+
+module ScrapeUnblocker
+  # Base class for every error raised by this library.
+  class Error < StandardError; end
+
+  # An error response returned by the ScrapeUnblocker API.
+  class APIError < Error
+    attr_reader :status_code, :body
+
+    def initialize(message, status_code:, body: nil)
+      super(message)
+      @status_code = status_code
+      @body = body
+    end
+  end
+
+  # The API key is missing, malformed, or not recognised (HTTP 401).
+  class AuthenticationError < APIError; end
+
+  # The request was rejected as invalid, e.g. a malformed URL (HTTP 400).
+  class InvalidRequestError < APIError; end
+
+  # The target site blocked every available bypass path (HTTP 403).
+  # Blocked calls are not billed.
+  class BlockedError < APIError; end
+
+  # Too many requests against your account in a short window (HTTP 429).
+  class RateLimitError < APIError; end
+
+  # The origin site returned a server-side outage page (HTTP 503).
+  class UpstreamOutageError < APIError; end
+
+  # ScrapeUnblocker returned an unexpected 5xx error.
+  class ServerError < APIError; end
+
+  # The request did not complete within the configured timeout.
+  class TimeoutError < Error; end
+
+  # The client could not reach the ScrapeUnblocker API.
+  class ConnectionError < Error; end
+
+  # Build a typed error from an HTTP status code and response body.
+  def self.error_for_status(status, body)
+    snippet = (body || "").strip.gsub(/\s+/, " ")
+    snippet = "#{snippet[0, 200]}..." if snippet.length > 200
+    base = {
+      400 => "Invalid request (bad URL or unsupported scheme)",
+      401 => "Authentication failed - check your API key",
+      403 => "Target blocked by bot protection on every bypass path",
+      429 => "Rate limited - too many requests",
+      503 => "Upstream origin returned a server-side outage page"
+    }.fetch(status, "API returned HTTP #{status}")
+    message = snippet.empty? ? base : "#{base}: #{snippet}"
+
+    klass =
+      case status
+      when 400 then InvalidRequestError
+      when 401 then AuthenticationError
+      when 403 then BlockedError
+      when 429 then RateLimitError
+      when 503 then UpstreamOutageError
+      else status >= 500 ? ServerError : APIError
+      end
+
+    klass.new(message, status_code: status, body: body)
+  end
+end

data/lib/scrapeunblocker/parsed_page.rb

@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+
+module ScrapeUnblocker
+  # Structured data extracted from a page (#get_parsed).
+  class ParsedPage
+    # @return [String, nil] what the API classified the page as, e.g. "product"
+    attr_reader :page_type
+    # @return [String, nil] how the data was extracted
+    attr_reader :source
+    # @return the extracted fields
+    attr_reader :data
+    # @return [Hash] the full JSON payload as returned by the API
+    attr_reader :raw
+
+    def initialize(page_type:, source:, data:, raw:)
+      @page_type = page_type
+      @source = source
+      @data = data
+      @raw = raw
+    end
+
+    def self.from_hash(payload)
+      inner = payload["data"].is_a?(Hash) ? payload["data"] : payload
+      new(
+        page_type: inner["page_type"],
+        source: inner["source"],
+        data: inner["data"],
+        raw: payload
+      )
+    end
+  end
+
+  # HTML plus the cookies and proxy that served it (#get_page_with_cookies).
+  class PageResult
+    attr_reader :html, :cookies, :proxy, :raw
+
+    def initialize(html:, cookies:, proxy:, raw:)
+      @html = html
+      @cookies = cookies
+      @proxy = proxy
+      @raw = raw
+    end
+
+    def self.from_hash(payload)
+      new(
+        html: payload["html"] || payload["page_source"] || payload["content"],
+        cookies: payload["cookies"],
+        proxy: payload["proxy"] || payload["proxy_address"],
+        raw: payload
+      )
+    end
+  end
+end

data/lib/scrapeunblocker/skyscanner.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+module ScrapeUnblocker
+  # Skyscanner plugin endpoints (flights, hotels, car hire).
+  class Skyscanner
+    # @api private
+    def initialize(client)
+      @client = client
+    end
+
+    def flight_locations(q, params = {})
+      @client.post_json("/flights/skyscanner-locations", { q: q }.merge(params))
+    end
+
+    def flights(params = {})
+      @client.post_json("/flights/skyscanner-quotes", params)
+    end
+
+    def hotel_locations(q, params = {})
+      @client.post_json("/hotels/skyscanner-locations", { q: q }.merge(params))
+    end
+
+    def hotels(params = {})
+      @client.post_json("/hotels/skyscanner-quotes", params)
+    end
+
+    def carhire_locations(q, params = {})
+      @client.post_json("/carhire/skyscanner-locations", { q: q }.merge(params))
+    end
+
+    def carhire(params = {})
+      @client.post_json("/carhire/skyscanner-quotes", params)
+    end
+  end
+end

data/lib/scrapeunblocker/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module ScrapeUnblocker
+  VERSION = "0.1.0"
+end

data/lib/scrapeunblocker.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+require_relative "scrapeunblocker/version"
+require_relative "scrapeunblocker/errors"
+require_relative "scrapeunblocker/parsed_page"
+require_relative "scrapeunblocker/skyscanner"
+require_relative "scrapeunblocker/client"
+
+# Official Ruby client for the ScrapeUnblocker web scraping API.
+#
+#   require "scrapeunblocker"
+#
+#   su = ScrapeUnblocker::Client.new  # reads SCRAPEUNBLOCKER_KEY
+#   html = su.get_page_source("https://example.com")
+#   product = su.get_parsed("https://www.amazon.com/dp/B08N5WRWNW")
+module ScrapeUnblocker
+end

metadata

@@ -0,0 +1,59 @@
+--- !ruby/object:Gem::Specification
+name: scrapeunblocker
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- ScrapeUnblocker
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2026-07-16 00:00:00.000000000 Z
+dependencies: []
+description: JS-rendered pages that bypass Cloudflare, DataDome, PerimeterX and Akamai,
+  plus Google SERP and Skyscanner flights/hotels/car-hire scraping as JSON.
+email:
+- support@scrapeunblocker.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE
+- README.md
+- lib/scrapeunblocker.rb
+- lib/scrapeunblocker/client.rb
+- lib/scrapeunblocker/errors.rb
+- lib/scrapeunblocker/parsed_page.rb
+- lib/scrapeunblocker/skyscanner.rb
+- lib/scrapeunblocker/version.rb
+homepage: https://scrapeunblocker.com
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://scrapeunblocker.com
+  source_code_uri: https://github.com/ScrapeUnblocker/scrapeunblocker-ruby
+  documentation_uri: https://developers.scrapeunblocker.com
+  changelog_uri: https://github.com/ScrapeUnblocker/scrapeunblocker-ruby/blob/main/CHANGELOG.md
+  bug_tracker_uri: https://github.com/ScrapeUnblocker/scrapeunblocker-ruby/issues
+  rubygems_mfa_required: 'true'
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.7.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.5.22
+signing_key:
+specification_version: 4
+summary: Official Ruby client for the ScrapeUnblocker web scraping API.
+test_files: []