webscraping_ai 3.2.0 → 4.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ea239c0ee1260cacb701ce616aef022672f3aa35ca5ec44e6db5348eee1e9fa4
-  data.tar.gz: f2a9af9da881dc8c24396847cbfa724a91f1b12da5b2eeab38b492913383d64b
+  metadata.gz: 9118cbe0e21d02653f6d56c66be8c05a21735241529782164f1e5352236d8567
+  data.tar.gz: 3e6d144a7dac8202e8e1405a612c1cacfda06301d17cb6c815ba35e6333237f7
 SHA512:
-  metadata.gz: 9d9d8624aa0e2159e6009e2ae3719b2cec9dce3a3d51883441159a8ac9ace76fa801c043bbf19950e001519d06226cc102fcdef746ec427648af1b380d963e84
-  data.tar.gz: 6afde37248d490b226cc41ffb41b972d4d6b427878e36ff0a267ea9e6959b080b2309f4e651151cf762fe2819ab52fe712ff5ae6becb2be1c7f5f1df788c5cd4
+  metadata.gz: af8e1bab5b5887224e6a0bae94ce178e2cc1a7405310fa60f4d11de621a6fcc1dbe28034b11247a29ff38cc33a63b7a3fc12908e88d54efe4500fe3cf617f801
+  data.tar.gz: 1e90885ed4a79632c94102863325eabfe5968cbe1901dec0f6a568526a416a3cf22550f8b8534927008c8aeef6e55c33372cdba6b50c62f74832cd74213879bd

data/CHANGELOG.md

@@ -0,0 +1,25 @@
+# Changelog
+
+All notable changes to this project will be documented in this file. This project follows [Semantic Versioning](https://semver.org/).
+
+## [4.0.0] - Unreleased
+
+### Changed
+
+- **Complete rewrite**: the gem is now a hand-written, idiomatic Ruby client rather than OpenAPI-generated code.
+- New unified entry point: `WebScrapingAI::Client.new(api_key: ...)` with one method per endpoint (`#html`, `#text`, `#selected`, `#selected_multiple`, `#question`, `#fields`, `#account`).
+- Switched HTTP layer from `typhoeus` to `faraday ~> 2.0`.
+- Minimum Ruby version is now `3.1`.
+
+### Removed
+
+- `WebScrapingAI::HTMLApi`, `WebScrapingAI::TextApi`, `WebScrapingAI::SelectedHTMLApi`, `WebScrapingAI::AIApi`, `WebScrapingAI::AccountApi` classes and all generated model classes. This is a hard break — see the README for the new API surface.
+- `typhoeus` runtime dependency.
+
+### Added
+
+- Typed error hierarchy: `BadRequestError`, `PaymentRequiredError`, `AuthenticationError`, `RateLimitError`, `ServerError`, `GatewayTimeoutError` (all `< WebScrapingAI::ApiError`), plus `TimeoutError` and `ConnectionError` for transport failures.
+- Module-level configuration: `WebScrapingAI.configure { |c| c.api_key = "..." }`.
+- `WEBSCRAPING_AI_API_KEY` environment variable picked up by default.
+- RSpec test suite with WebMock-based stubs.
+- GitHub Actions workflows for CI and RubyGems trusted publishing on release.

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) WebScraping.AI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -1,128 +1,153 @@
-# webscraping_ai
+# WebScraping.AI Ruby Client
 
-WebScrapingAI - the Ruby gem for the WebScraping.AI
+Official Ruby client for the [WebScraping.AI](https://webscraping.ai) API. Provides LLM-powered web scraping with Chromium JavaScript rendering, rotating proxies, and built-in HTML parsing.
 
-WebScraping.AI scraping API provides LLM-powered tools with Chromium JavaScript rendering, rotating proxies, and built-in HTML parsing.
-
-This SDK is automatically generated by the [OpenAPI Generator](https://openapi-generator.tech) project:
-
-- API version: 3.2.0
-- Package version: 3.2.0
-- Generator version: 7.11.0
-- Build package: org.openapitools.codegen.languages.RubyClientCodegen
-For more information, please visit [https://webscraping.ai](https://webscraping.ai)
+[![Gem Version](https://badge.fury.io/rb/webscraping_ai.svg)](https://rubygems.org/gems/webscraping_ai)
 
 ## Installation
 
-### Build a gem
-
-To build the Ruby code into a gem:
-
-```shell
-gem build webscraping_ai.gemspec
+```ruby
+# Gemfile
+gem "webscraping_ai", "~> 4.0"
 ```
 
-Then either install the gem locally:
+Or:
 
-```shell
-gem install ./webscraping_ai-3.2.0.gem
+```bash
+gem install webscraping_ai
 ```
 
-(for development, run `gem install --dev ./webscraping_ai-3.2.0.gem` to install the development dependencies)
+Requires Ruby 3.1+.
 
-or publish the gem to a gem hosting service, e.g. [RubyGems](https://rubygems.org/).
+## Quick start
 
-Finally add this to the Gemfile:
+```ruby
+require "webscraping_ai"
+
+client = WebScrapingAI::Client.new(api_key: ENV.fetch("WEBSCRAPING_AI_API_KEY"))
 
-    gem 'webscraping_ai', '~> 3.2.0'
+# Page HTML
+html = client.html("https://example.com", js: true)
 
-### Install from Git
+# Visible text
+text = client.text("https://example.com")
 
-If the Ruby gem is hosted at a git repository: https://github.com/webscraping-ai/webscraping-ai-ruby, then add the following in the Gemfile:
+# CSS-selected fragment
+title = client.selected("https://example.com", selector: "h1")
 
-    gem 'webscraping_ai', :git => 'https://github.com/webscraping-ai/webscraping-ai-ruby.git'
+# Multiple selectors at once
+fragments = client.selected_multiple("https://example.com", selectors: ["h1", ".price"])
 
-### Include the Ruby code directly
+# Ask the LLM a question about the page
+answer = client.question("https://example.com", question: "What is the main product?")
 
-Include the Ruby code directly using `-I` as follows:
+# Extract structured fields with the LLM
+data = client.fields(
+  "https://example.com",
+  fields: {
+    title: "Main product title",
+    price: "Current product price",
+    description: "Full product description"
+  }
+)
 
-```shell
-ruby -Ilib script.rb
+# Check your account quota
+info = client.account
+# => { "remaining_api_calls" => 200_000, "resets_at" => 1_617_073_667, "remaining_concurrency" => 100 }
 ```
 
-## Getting Started
+## Configuration
 
-Please follow the [installation](#installation) procedure and then run the following code:
+Configure globally once, then create clients without arguments:
 
 ```ruby
-# Load the gem
-require 'webscraping_ai'
-
-# Setup authorization
 WebScrapingAI.configure do |config|
-  # Configure API key authorization: api_key
-  config.api_key['api_key'] = 'YOUR API KEY'
-  # Uncomment the following line to set a prefix for the API key, e.g. 'Bearer' (defaults to nil)
-  # config.api_key_prefix['api_key'] = 'Bearer'
+  config.api_key      = ENV.fetch("WEBSCRAPING_AI_API_KEY")
+  config.timeout      = 60   # seconds, total request timeout
+  config.open_timeout = 10   # seconds, connection timeout
 end
 
-api_instance = WebScrapingAI::AIApi.new
-url = 'https://example.com' # String | URL of the target page.
-fields = { key: { key: 'inner_example'}} # Hash<String, String> | Object describing fields to extract from the page and their descriptions
-opts = {
-  headers: { key: { key: 'inner_example'}}, # Hash<String, String> | HTTP headers to pass to the target page. Can be specified either via a nested query parameter (...&headers[One]=value1&headers=[Another]=value2) or as a JSON encoded object (...&headers={\"One\": \"value1\", \"Another\": \"value2\"}).
-  timeout: 10000, # Integer | Maximum web page retrieval time in ms. Increase it in case of timeout errors (10000 by default, maximum is 30000).
-  js: true, # Boolean | Execute on-page JavaScript using a headless browser (true by default).
-  js_timeout: 2000, # Integer | Maximum JavaScript rendering time in ms. Increase it in case if you see a loading indicator instead of data on the target page.
-  wait_for: 'wait_for_example', # String | CSS selector to wait for before returning the page content. Useful for pages with dynamic content loading. Overrides js_timeout.
-  proxy: 'datacenter', # String | Type of proxy, use residential proxies if your site restricts traffic from datacenters (datacenter by default). Note that residential proxy requests are more expensive than datacenter, see the pricing page for details.
-  country: 'us', # String | Country of the proxy to use (US by default).
-  custom_proxy: 'custom_proxy_example', # String | Your own proxy URL to use instead of our built-in proxy pool in \"http://user:password@host:port\" format (<a target=\"_blank\" href=\"https://webscraping.ai/proxies/smartproxy\">Smartproxy</a> for example).
-  device: 'desktop', # String | Type of device emulation.
-  error_on_404: false, # Boolean | Return error on 404 HTTP status on the target page (false by default).
-  error_on_redirect: false, # Boolean | Return error on redirect on the target page (false by default).
-  js_script: 'document.querySelector('button').click();' # String | Custom JavaScript code to execute on the target page.
-}
+client = WebScrapingAI::Client.new
+```
+
+The gem also reads `WEBSCRAPING_AI_API_KEY` from the environment automatically.
 
-begin
-  #Extract structured data fields from a web page
-  result = api_instance.get_fields(url, fields, opts)
-  p result
-rescue WebScrapingAI::ApiError => e
-  puts "Exception when calling AIApi->get_fields: #{e}"
-end
+Per-instance overrides:
 
+```ruby
+client = WebScrapingAI::Client.new(
+  api_key: "...",
+  timeout: 90,
+  base_url: "https://api.webscraping.ai"
+)
 ```
 
-## Documentation for API Endpoints
+## Endpoints and options
 
-All URIs are relative to *https://api.webscraping.ai*
+All page-fetching endpoints accept these common options (passed as keyword arguments):
 
-Class | Method | HTTP request | Description
------------- | ------------- | ------------- | -------------
-*WebScrapingAI::AIApi* | [**get_fields**](docs/AIApi.md#get_fields) | **GET** /ai/fields | Extract structured data fields from a web page
-*WebScrapingAI::AIApi* | [**get_question**](docs/AIApi.md#get_question) | **GET** /ai/question | Get an answer to a question about a given web page
-*WebScrapingAI::AccountApi* | [**account**](docs/AccountApi.md#account) | **GET** /account | Information about your account calls quota
-*WebScrapingAI::HTMLApi* | [**get_html**](docs/HTMLApi.md#get_html) | **GET** /html | Page HTML by URL
-*WebScrapingAI::SelectedHTMLApi* | [**get_selected**](docs/SelectedHTMLApi.md#get_selected) | **GET** /selected | HTML of a selected page area by URL and CSS selector
-*WebScrapingAI::SelectedHTMLApi* | [**get_selected_multiple**](docs/SelectedHTMLApi.md#get_selected_multiple) | **GET** /selected-multiple | HTML of multiple page areas by URL and CSS selectors
-*WebScrapingAI::TextApi* | [**get_text**](docs/TextApi.md#get_text) | **GET** /text | Page text by URL
+| Option | Type | Default | Description |
+| --- | --- | --- | --- |
+| `headers` | `Hash` | — | HTTP headers to send to the target page (e.g. `{ "Cookie" => "session=..." }`) |
+| `timeout` | `Integer` | `10000` | Page retrieval timeout in ms (1–30000) |
+| `js` | `Boolean` | `true` | Execute on-page JavaScript via headless Chromium |
+| `js_timeout` | `Integer` | `2000` | JS rendering timeout in ms (1–20000) |
+| `wait_for` | `String` | — | CSS selector to wait for before returning (overrides `js_timeout`) |
+| `proxy` | `String` | `"datacenter"` | One of `datacenter`, `residential`, `stealth` |
+| `country` | `String` | `"us"` | Proxy country: `us`, `gb`, `de`, `it`, `fr`, `ca`, `es`, `ru`, `jp`, `kr`, `in`, `hk`, `tr` |
+| `custom_proxy` | `String` | — | Your own proxy in `http://user:pass@host:port` form |
+| `device` | `String` | `"desktop"` | One of `desktop`, `mobile`, `tablet` |
+| `error_on_404` | `Boolean` | `false` | Raise an error if the target page returns 404 |
+| `error_on_redirect` | `Boolean` | `false` | Raise an error if the target page redirects |
+| `js_script` | `String` | — | Custom JS to execute on the page |
 
+Endpoint-specific options:
 
-## Documentation for Models
+- `#html` — `return_script_result` (`Boolean`), `format` (`"json"`/`"text"`)
+- `#text` — `text_format` (`"plain"`/`"xml"`/`"json"`), `return_links` (`Boolean`, only with `text_format: "json"`)
+- `#selected` — `selector` (`String`), `format` (`"json"`/`"text"`)
+- `#selected_multiple` — `selectors` (`Array<String>` or single `String`)
+- `#question` — `question` (`String`, required), `format` (`"json"`/`"text"`)
+- `#fields` — `fields` (`Hash<String, String>`, required) — keys are field names, values are descriptions
 
- - [WebScrapingAI::Account](docs/Account.md)
- - [WebScrapingAI::Error](docs/Error.md)
+Returns: `String` for HTML/text responses, `Hash`/`Array` for JSON responses.
 
+## Error handling
 
-## Documentation for Authorization
+All API errors inherit from `WebScrapingAI::ApiError` and expose `#status`, `#message`, `#status_code`, `#status_message`, `#body`, and `#response_body`.
 
+```ruby
+begin
+  client.html("https://example.com")
+rescue WebScrapingAI::RateLimitError => e
+  # 429 — too many concurrent requests
+  sleep 1 and retry
+rescue WebScrapingAI::PaymentRequiredError => e
+  # 402 — out of API credits
+rescue WebScrapingAI::AuthenticationError => e
+  # 403 — wrong API key
+rescue WebScrapingAI::BadRequestError => e
+  # 400 — invalid parameters
+rescue WebScrapingAI::ServerError => e
+  # 500 — target page returned a non-2xx code, or unexpected error.
+  # e.status_code / e.status_message expose the target page's response.
+rescue WebScrapingAI::GatewayTimeoutError => e
+  # 504 — page took longer than `timeout` ms to load. Try a higher `timeout:`.
+rescue WebScrapingAI::TimeoutError => e
+  # Client-side: the HTTP request exceeded `Client#timeout`.
+rescue WebScrapingAI::ConnectionError => e
+  # Network failure before a response was received.
+end
+```
 
-Authentication schemes defined for the API:
-### api_key
+## Development
 
+```bash
+bin/setup        # bundle install
+bundle exec rspec
+bundle exec rubocop
+```
 
-- **Type**: API key
-- **API key parameter name**: api_key
-- **Location**: URL query string
+## License
 
+MIT — see [LICENSE](LICENSE).

data/lib/webscraping_ai/client.rb

@@ -0,0 +1,130 @@
+require "faraday"
+require "json"
+
+module WebScrapingAI
+  class Client
+    PROXY_TYPES = %w[datacenter residential stealth].freeze
+    COUNTRIES = %w[us gb de it fr ca es ru jp kr in hk tr].freeze
+    DEVICES = %w[desktop mobile tablet].freeze
+    TEXT_FORMATS = %w[plain xml json].freeze
+    FORMATS = %w[json text].freeze
+
+    PAGE_FETCH_OPTIONS = %i[
+      headers timeout js js_timeout wait_for proxy country
+      custom_proxy device error_on_404 error_on_redirect js_script
+    ].freeze
+
+    attr_reader :configuration
+
+    def initialize(api_key: nil, base_url: nil, timeout: nil, open_timeout: nil, adapter: nil, user_agent: nil)
+      global = WebScrapingAI.configuration
+      @configuration = Configuration.new.tap do |c|
+        c.api_key = api_key || global.api_key
+        c.base_url = base_url || global.base_url
+        c.timeout = timeout || global.timeout
+        c.open_timeout = open_timeout || global.open_timeout
+        c.adapter = adapter || global.adapter
+        c.user_agent = user_agent || global.user_agent
+      end
+
+      return unless @configuration.api_key.nil? || @configuration.api_key.to_s.empty?
+
+      raise ConfigurationError,
+            "api_key is required (pass api_key: or set WebScrapingAI.configure { |c| c.api_key = ... })"
+    end
+
+    # GET /ai/question — returns the LLM's answer about the page.
+    # Returns a String by default, or a Hash when format: "json".
+    def question(url, question:, **opts)
+      get("/ai/question", url: url, question: question, **opts.slice(*PAGE_FETCH_OPTIONS, :format))
+    end
+
+    # GET /ai/fields — extracts the named fields from the page.
+    # `fields` is a Hash of { field_name => description }. Returns a Hash.
+    def fields(url, fields:, **opts)
+      get("/ai/fields", url: url, fields: fields, **opts.slice(*PAGE_FETCH_OPTIONS))
+    end
+
+    # GET /html — returns the full page HTML as a String.
+    def html(url, **opts)
+      get("/html", url: url, **opts.slice(*PAGE_FETCH_OPTIONS, :return_script_result, :format))
+    end
+
+    # GET /text — returns the visible text content of the page.
+    # Returns a String when text_format is "plain"/"xml" (default), or a Hash when text_format: "json".
+    def text(url, **opts)
+      get("/text", url: url, **opts.slice(*PAGE_FETCH_OPTIONS, :text_format, :return_links))
+    end
+
+    # GET /selected — returns HTML of the element matching `selector` as a String.
+    def selected(url, selector: nil, **opts)
+      get("/selected", url: url, selector: selector, **opts.slice(*PAGE_FETCH_OPTIONS, :format))
+    end
+
+    # GET /selected-multiple — returns an Array of HTML strings, one per selector.
+    def selected_multiple(url, selectors:, **opts)
+      get("/selected-multiple", url: url, selectors: Array(selectors), **opts.slice(*PAGE_FETCH_OPTIONS))
+    end
+
+    # GET /account — returns Hash with remaining_api_calls, resets_at, remaining_concurrency, email.
+    def account
+      get("/account")
+    end
+
+    private
+
+    def connection
+      @connection ||= Faraday.new(url: configuration.base_url) do |conn|
+        conn.options.timeout = configuration.timeout
+        conn.options.open_timeout = configuration.open_timeout
+        conn.options.params_encoder = QueryEncoder
+        conn.headers["User-Agent"] = configuration.user_agent
+        conn.headers["Accept"] = "application/json, text/html, text/xml, text/plain"
+        conn.adapter(configuration.adapter || Faraday.default_adapter)
+      end
+    end
+
+    def get(path, **params)
+      response = connection.get(path) do |req|
+        req.params = params.merge(api_key: configuration.api_key)
+      end
+      handle_response(response)
+    rescue Faraday::TimeoutError => e
+      raise TimeoutError, e.message
+    rescue Faraday::ConnectionFailed => e
+      raise ConnectionError, e.message
+    end
+
+    def handle_response(response)
+      return parse_body(response) if response.status.between?(200, 299)
+
+      error_class = STATUS_TO_ERROR.fetch(response.status, ApiError)
+      data = safe_parse_json(response.body) || {}
+      raise error_class.new(
+        message: data["message"] || "HTTP #{response.status}",
+        status: response.status,
+        status_code: data["status_code"],
+        status_message: data["status_message"],
+        body: data["body"],
+        response_body: response.body
+      )
+    end
+
+    def parse_body(response)
+      content_type = response.headers["content-type"].to_s
+      if content_type.include?("application/json")
+        JSON.parse(response.body)
+      else
+        response.body
+      end
+    end
+
+    def safe_parse_json(body)
+      return nil if body.nil? || body.empty?
+
+      JSON.parse(body)
+    rescue JSON::ParserError
+      nil
+    end
+  end
+end