zenrows 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: af4d111e269c080e71f047c74acec3a319f05e0b6c800cb7067b4d05a23cc209
+  data.tar.gz: 373c8b95c4f311141767d750a509e88ef74dc2e18c947a703b9868c5cce97ad4
+SHA512:
+  metadata.gz: 81b92b61f13e82a6b2462e790df7f009f88d75f7d9cac822dfd5e685fab0c85b85898f50f3ce67b241622609ff02ebf2b98896e0b9a2ed870f41d1ea2231138c
+  data.tar.gz: 9740ef08c21c4d3701795786886df16183d188a0cf6d5fc7d04bc0a2a46b8f00693ec98fecfd4925bf9a2cd69ab9dca93add9e4c54f2631c630f2698444f912a

data/.standard.yml

@@ -0,0 +1,8 @@
+# Standard RB configuration
+# https://github.com/standardrb/standard
+
+ruby_version: 3.1
+
+ignore:
+  - "vendor/**/*"
+  - "tmp/**/*"

data/.yardopts

@@ -0,0 +1,10 @@
+--markup markdown
+--charset utf-8
+--readme README.md
+--title "Zenrows Ruby Client"
+--protected
+--no-private
+lib/**/*.rb
+-
+LICENSE.txt
+CHANGELOG.md

data/CHANGELOG.md

@@ -0,0 +1,28 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.1.0] - 2025-12-25
+
+### Added
+
+- Initial release
+- Multi-backend architecture with http.rb as primary adapter
+- Proxy mode support (ZenRows superproxy)
+- JavaScript rendering support
+- Premium proxy and geolocation options
+- JavaScript Instructions DSL for browser automation
+- Screenshot support (full page and element)
+- Session persistence
+- Block resources option
+- Wait and wait_for parameters
+- Custom headers support
+- YARD documentation
+- Minitest test suite
+- Standard RB linting
+- Optional Rails integration with ActiveSupport::Duration support

data/CLAUDE.md

@@ -0,0 +1,63 @@
+# CLAUDE.md
+
+Ruby gem for ZenRows web scraping proxy. Multi-backend HTTP client (http.rb primary).
+
+## Context7 MCP
+
+Use Context7 for docs lookup:
+- `/websites/guides_rubygems` - RubyGems packaging
+- `/docs.zenrows.com-4bed007/llmstxt` - ZenRows API
+- `/websites/code_claude_en` - Claude Code CLI
+
+## Commands
+
+```bash
+bundle install          # Install dependencies
+bundle exec rake test   # Run tests
+bundle exec rubocop     # Lint code
+bundle exec yard doc    # Generate docs
+bundle exec rake build  # Build gem
+```
+
+## Architecture
+
+```
+lib/zenrows/
+├── version.rb           # Gem version
+├── configuration.rb     # Global config (api_key, host, port)
+├── client.rb            # Main client, returns HTTP instances
+├── proxy.rb             # Proxy URL builder (options in username)
+├── js_instructions.rb   # DSL for browser automation
+├── errors.rb            # Custom exceptions
+├── railtie.rb           # Optional Rails integration
+└── backends/
+    ├── base.rb          # Backend interface
+    └── http_rb.rb       # http.rb adapter (primary)
+```
+
+## Usage
+
+```ruby
+Zenrows.configure do |c|
+  c.api_key = 'YOUR_KEY'
+  c.host = 'superproxy.zenrows.com'
+  c.port = 1337
+end
+
+client = Zenrows::Client.new
+http = client.http(js_render: true, premium_proxy: true)
+response = http.get('https://example.com', ssl_context: client.ssl_context)
+```
+
+## Key Options
+
+| Option | Type | Description |
+|--------|------|-------------|
+| `js_render` | Boolean | Enable headless browser |
+| `premium_proxy` | Boolean | Use residential IPs |
+| `proxy_country` | String | Country code (us, gb, de) |
+| `wait` | Integer | Wait time in ms |
+| `wait_for` | String | CSS selector to wait for |
+| `js_instructions` | Array/JSON | Browser automation |
+| `session_id` | Bool/String | Sticky session |
+| `screenshot` | Boolean | Take screenshot |

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Fourthwall
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,174 @@
+# Zenrows
+
+Ruby client for [ZenRows](https://www.zenrows.com/) web scraping proxy. Multi-backend HTTP client with http.rb as primary adapter.
+
+## Installation
+
+Add to your Gemfile:
+
+```ruby
+gem 'zenrows'
+```
+
+Then run:
+
+```bash
+bundle install
+```
+
+## Configuration
+
+```ruby
+Zenrows.configure do |config|
+  config.api_key = 'YOUR_API_KEY'
+  config.host = 'superproxy.zenrows.com'  # default
+  config.port = 1337                       # default
+  config.connect_timeout = 5               # seconds
+  config.read_timeout = 180                # seconds
+end
+```
+
+## Usage
+
+### Basic Request
+
+```ruby
+client = Zenrows::Client.new
+http = client.http(js_render: true, premium_proxy: true)
+response = http.get('https://example.com', ssl_context: client.ssl_context)
+
+puts response.body
+puts response.status
+```
+
+### With Options
+
+```ruby
+http = client.http(
+  js_render: true,           # Enable headless browser
+  premium_proxy: true,       # Use residential IPs
+  proxy_country: 'us',       # Geolocation
+  wait: 5000,                # Wait 5 seconds after load
+  wait_for: '.content',      # Wait for CSS selector
+  session_id: true           # Sticky session
+)
+```
+
+### JavaScript Instructions
+
+Automate browser interactions:
+
+```ruby
+instructions = Zenrows::JsInstructions.build do
+  wait_for '.login-form'
+  fill '#email', 'user@example.com'
+  fill '#password', 'secret123'
+  click '#submit'
+  wait 2000
+  scroll_to :bottom
+  wait_for '.results'
+end
+
+http = client.http(js_render: true, js_instructions: instructions)
+response = http.get(url, ssl_context: client.ssl_context)
+```
+
+Available instructions:
+- `click(selector)` - Click element
+- `wait(ms)` - Wait duration
+- `wait_for(selector)` - Wait for element
+- `wait_event(event)` - networkidle, load, domcontentloaded
+- `fill(selector, value)` - Fill input
+- `check(selector)` / `uncheck(selector)` - Checkboxes
+- `select_option(selector, value)` - Dropdowns
+- `scroll_y(pixels)` / `scroll_x(pixels)` - Scroll
+- `scroll_to(:bottom)` / `scroll_to(:top)` - Scroll to position
+- `evaluate(js_code)` - Execute JavaScript
+- `frame_*` variants for iframe interactions
+
+### Screenshots
+
+```ruby
+http = client.http(
+  js_render: true,
+  screenshot: true,           # Take screenshot
+  screenshot_fullpage: true,  # Full page
+  json_response: true         # Get JSON with screenshot data
+)
+```
+
+### Block Resources
+
+Speed up requests by blocking unnecessary resources:
+
+```ruby
+http = client.http(
+  js_render: true,
+  block_resources: 'image,media,font'
+)
+```
+
+## Options Reference
+
+| Option | Type | Description |
+|--------|------|-------------|
+| `js_render` | Boolean | Enable JavaScript rendering |
+| `premium_proxy` | Boolean | Use residential proxies |
+| `proxy_country` | String | Country code (us, gb, de, etc.) |
+| `wait` | Integer/Boolean | Wait time in ms (true = 15000) |
+| `wait_for` | String | CSS selector to wait for |
+| `session_id` | Boolean/String | Session persistence |
+| `window_height` | Integer | Browser window height |
+| `window_width` | Integer | Browser window width |
+| `js_instructions` | Array/String | Browser automation |
+| `json_response` | Boolean | Return JSON instead of HTML |
+| `screenshot` | Boolean | Take screenshot |
+| `screenshot_fullpage` | Boolean | Full page screenshot |
+| `screenshot_selector` | String | Screenshot specific element |
+| `block_resources` | String | Block resources (image,media,font) |
+| `headers` | Hash | Custom HTTP headers |
+
+## Error Handling
+
+```ruby
+begin
+  response = http.get(url, ssl_context: client.ssl_context)
+rescue Zenrows::ConfigurationError => e
+  # Missing or invalid configuration
+rescue Zenrows::RateLimitError => e
+  sleep(e.retry_after || 60)
+  retry
+rescue Zenrows::BotDetectedError => e
+  # Try with premium proxy
+  http = client.http(premium_proxy: true, proxy_country: 'us')
+  retry
+rescue Zenrows::WaitTimeError => e
+  # Wait time exceeded 3 minutes
+rescue Zenrows::TimeoutError => e
+  # Request timed out
+end
+```
+
+## Rails Integration
+
+The gem automatically integrates with Rails when detected:
+- Uses Rails.logger by default
+- Supports ActiveSupport::Duration for wait times
+
+```ruby
+# In Rails, you can use duration objects
+http = client.http(wait: 5.seconds)
+```
+
+## Development
+
+```bash
+bundle install
+bundle exec rake test      # Run tests
+bundle exec standardrb     # Lint code
+bundle exec yard doc       # Generate docs
+```
+
+## License
+
+MIT License. See [LICENSE.txt](LICENSE.txt).

data/Rakefile

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "minitest/test_task"
+
+Minitest::TestTask.create(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.warning = false
+  t.test_globs = ["test/**/*_test.rb"]
+end
+
+require "standard/rake"
+
+task default: %i[test standard]

data/lib/zenrows/backends/base.rb

@@ -0,0 +1,95 @@
+# frozen_string_literal: true
+
+module Zenrows
+  module Backends
+    # Abstract base class for HTTP backends
+    #
+    # Backends are responsible for building configured HTTP clients
+    # that route through the ZenRows proxy.
+    #
+    # @abstract Subclass and override {#build_client} to implement
+    # @author Ernest Bursa
+    # @since 0.1.0
+    # @api public
+    class Base
+      # @return [Zenrows::Proxy] Proxy configuration builder
+      attr_reader :proxy
+
+      # @return [Zenrows::Configuration] Configuration instance
+      attr_reader :config
+
+      # @param proxy [Zenrows::Proxy] Proxy configuration builder
+      # @param config [Zenrows::Configuration] Configuration instance
+      def initialize(proxy:, config:)
+        @proxy = proxy
+        @config = config
+      end
+
+      # Build a configured HTTP client
+      #
+      # @param options [Hash] Request options
+      # @return [Object] Configured HTTP client
+      # @raise [NotImplementedError] if not implemented by subclass
+      def build_client(options = {})
+        raise NotImplementedError, "#{self.class}#build_client must be implemented"
+      end
+
+      # Build SSL context for proxy connections
+      #
+      # @return [OpenSSL::SSL::SSLContext] SSL context with verification disabled
+      def ssl_context
+        require "openssl"
+        ctx = OpenSSL::SSL::SSLContext.new
+        ctx.verify_mode = OpenSSL::SSL::VERIFY_NONE
+        ctx
+      end
+
+      # Calculate appropriate timeout based on options
+      #
+      # @param options [Hash] Request options
+      # @return [Hash] Timeout configuration with :connect and :read
+      def calculate_timeouts(options = {})
+        connect = config.connect_timeout
+        read = config.read_timeout
+
+        # Add time for JS rendering
+        read += 15 if options[:js_render]
+
+        # Add buffer for wait time
+        if options[:wait]
+          wait_seconds = normalize_wait_seconds(options[:wait])
+          read += wait_seconds + 20
+        end
+
+        # Add time for screenshots
+        read += 10 if options[:screenshot] || options[:screenshot_fullpage]
+
+        # Add time for JS instructions
+        if options[:js_instructions]
+          instructions = options[:js_instructions]
+          count = instructions.is_a?(Array) ? instructions.size : 1
+          read += count * 5
+        end
+
+        {connect: connect, read: read}
+      end
+
+      private
+
+      # Normalize wait value to seconds
+      #
+      # @param wait [Boolean, Integer, Object] Wait value
+      # @return [Integer] Wait time in seconds
+      def normalize_wait_seconds(wait)
+        case wait
+        when true then 15
+        when Integer then wait / 1000
+        when ->(w) { w.respond_to?(:to_i) && w.respond_to?(:parts) }
+          wait.to_i
+        else
+          (wait.to_i / 1000.0).ceil
+        end
+      end
+    end
+  end
+end

data/lib/zenrows/backends/http_rb.rb

@@ -0,0 +1,59 @@
+# frozen_string_literal: true
+
+require "http"
+
+module Zenrows
+  module Backends
+    # HTTP.rb backend adapter
+    #
+    # Uses the http.rb gem to build configured HTTP clients
+    # that route through the ZenRows proxy.
+    #
+    # @example Basic usage
+    #   backend = Zenrows::Backends::HttpRb.new(proxy: proxy, config: config)
+    #   http = backend.build_client(js_render: true)
+    #   response = http.get(url, ssl_context: backend.ssl_context)
+    #
+    # @author Ernest Bursa
+    # @since 0.1.0
+    # @api public
+    class HttpRb < Base
+      # Build a configured HTTP client
+      #
+      # @param options [Hash] Request options
+      # @option options [Boolean] :js_render Enable JavaScript rendering
+      # @option options [Boolean] :premium_proxy Use residential proxies
+      # @option options [String] :proxy_country Country code
+      # @option options [Boolean, Integer] :wait Wait time
+      # @option options [String] :wait_for CSS selector to wait for
+      # @option options [Hash] :headers Custom HTTP headers
+      # @return [HTTP::Client] Configured HTTP client
+      def build_client(options = {})
+        opts = options.dup
+        headers = opts.delete(:headers) || {}
+
+        # Enable custom_headers if we have headers
+        opts[:custom_headers] = true if headers.any?
+
+        # Get proxy configuration
+        proxy_config = proxy.build(opts)
+
+        # Calculate timeouts
+        timeouts = calculate_timeouts(opts)
+
+        # Build HTTP client
+        client = HTTP
+          .timeout(connect: timeouts[:connect], read: timeouts[:read])
+          .headers(headers)
+
+        # Configure proxy
+        client.via(
+          proxy_config[:host],
+          proxy_config[:port],
+          proxy_config[:username],
+          proxy_config[:password]
+        )
+      end
+    end
+  end
+end

data/lib/zenrows/client.rb

@@ -0,0 +1,159 @@
+# frozen_string_literal: true
+
+module Zenrows
+  # Main client for ZenRows proxy
+  #
+  # The client builds configured HTTP clients that route through
+  # the ZenRows proxy with specified options.
+  #
+  # @example Basic usage
+  #   Zenrows.configure do |c|
+  #     c.api_key = 'YOUR_API_KEY'
+  #   end
+  #
+  #   client = Zenrows::Client.new
+  #   http = client.http(js_render: true)
+  #   response = http.get('https://example.com', ssl_context: client.ssl_context)
+  #
+  # @example With custom configuration
+  #   client = Zenrows::Client.new(api_key: 'KEY', host: 'proxy.zenrows.com')
+  #   http = client.http(premium_proxy: true, proxy_country: 'us')
+  #
+  # @author Ernest Bursa
+  # @since 0.1.0
+  # @api public
+  class Client
+    # @return [Configuration] Client configuration
+    attr_reader :config
+
+    # @return [Proxy] Proxy builder instance
+    attr_reader :proxy
+
+    # @return [Backends::Base] HTTP backend instance
+    attr_reader :backend
+
+    # Initialize a new client
+    #
+    # @param api_key [String, nil] Override API key from global config
+    # @param host [String, nil] Override proxy host
+    # @param port [Integer, nil] Override proxy port
+    # @param backend [Symbol] Backend to use (:http_rb)
+    # @raise [ConfigurationError] if api_key is not configured
+    def initialize(api_key: nil, host: nil, port: nil, backend: nil)
+      @config = build_config(api_key: api_key, host: host, port: port, backend: backend)
+      @config.validate!
+
+      @proxy = Proxy.new(
+        api_key: @config.api_key,
+        host: @config.host,
+        port: @config.port
+      )
+
+      @backend = build_backend
+    end
+
+    # Build a configured HTTP client
+    #
+    # @param options [Hash] Request options
+    # @option options [Boolean] :js_render Enable JavaScript rendering
+    # @option options [Boolean] :premium_proxy Use residential proxies
+    # @option options [String] :proxy_country Country code (us, gb, de, etc.)
+    # @option options [Boolean, Integer] :wait Wait time (true=15s, Integer=ms)
+    # @option options [String] :wait_for CSS selector to wait for
+    # @option options [Boolean, String, Integer] :session_id Session persistence
+    # @option options [Integer] :window_height Browser window height
+    # @option options [Integer] :window_width Browser window width
+    # @option options [Array, String] :js_instructions JavaScript instructions
+    # @option options [Boolean] :json_response Return JSON instead of HTML
+    # @option options [Boolean] :screenshot Take screenshot
+    # @option options [Boolean] :screenshot_fullpage Full page screenshot
+    # @option options [String] :screenshot_selector Screenshot specific element
+    # @option options [Hash] :headers Custom HTTP headers
+    # @option options [String] :block_resources Block resources (image,media,font)
+    # @return [HTTP::Client] Configured HTTP client ready for requests
+    #
+    # @example Basic request
+    #   http = client.http(js_render: true)
+    #   response = http.get(url, ssl_context: client.ssl_context)
+    #
+    # @example With premium proxy and country
+    #   http = client.http(premium_proxy: true, proxy_country: 'us')
+    #   response = http.get(url, ssl_context: client.ssl_context)
+    def http(options = {})
+      backend.build_client(options)
+    end
+
+    # Get SSL context for proxy connections
+    #
+    # ZenRows proxy requires SSL verification to be disabled.
+    #
+    # @return [OpenSSL::SSL::SSLContext] SSL context
+    #
+    # @example
+    #   http = client.http(js_render: true)
+    #   response = http.get(url, ssl_context: client.ssl_context)
+    def ssl_context
+      backend.ssl_context
+    end
+
+    # Get proxy configuration for given options
+    #
+    # @param options [Hash] Proxy options
+    # @return [Hash] Proxy configuration with :host, :port, :username, :password
+    def proxy_config(options = {})
+      proxy.build(options)
+    end
+
+    # Get proxy URL for given options
+    #
+    # @param options [Hash] Proxy options
+    # @return [String] Proxy URL
+    def proxy_url(options = {})
+      proxy.build_url(options)
+    end
+
+    private
+
+    # Build configuration from params and global config
+    #
+    # @param api_key [String, nil] Override API key
+    # @param host [String, nil] Override host
+    # @param port [Integer, nil] Override port
+    # @param backend [Symbol, nil] Override backend
+    # @return [Configuration] Configuration instance
+    def build_config(api_key:, host:, port:, backend:)
+      cfg = Configuration.new
+
+      # Start with global config values
+      global = Zenrows.configuration
+      cfg.api_key = global.api_key
+      cfg.host = global.host
+      cfg.port = global.port
+      cfg.connect_timeout = global.connect_timeout
+      cfg.read_timeout = global.read_timeout
+      cfg.backend = global.backend
+      cfg.logger = global.logger
+
+      # Override with provided values
+      cfg.api_key = api_key if api_key
+      cfg.host = host if host
+      cfg.port = port if port
+      cfg.backend = backend if backend
+
+      cfg
+    end
+
+    # Build backend instance based on configuration
+    #
+    # @return [Backends::Base] Backend instance
+    # @raise [ConfigurationError] if backend is not supported
+    def build_backend
+      case config.backend
+      when :http_rb
+        Backends::HttpRb.new(proxy: proxy, config: config)
+      else
+        raise ConfigurationError, "Unsupported backend: #{config.backend}. Use :http_rb"
+      end
+    end
+  end
+end