webscraping_ai 4.1.0 → 4.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cd7230f48fdc600563c0e7fe84ba65411f6363007854a9bdaae6b2ee6bfbf17d
-  data.tar.gz: a815bbc4645cd9eb23aba4e9cdafff0ac1c420f3a230580e002a59f7f19b28f6
+  metadata.gz: bd048e4bbb3a0bec22e8dac6616be20895e1c78761773295f164746985468261
+  data.tar.gz: 97972326da495b82684af9a58c95f9d0c3220d1b02559b54b1211381d5e16727
 SHA512:
-  metadata.gz: 8266c1f95d1432a37f817e3308e1c99c1795fba17fc105c4ef703561e89f84104f0d2734fda99dded525000baecd346d00c2ac3f0b838e251807d3651d20fae6
-  data.tar.gz: 6d075d815694065fde5b830d577100c8c45c9943798689ecc50b9959120d0486b10ede96b77344fb0803b8094f2855cf60d72237161b07a480e31b4531c62b1c
+  metadata.gz: 26d0348149011fcdb606717b7e4c8c178e9f21f544c0f720842678411c900dbecc673ad3046f0e0847704e818bdd6cee4aa9135e9c1e657bddd4e0cd109a1b82
+  data.tar.gz: f02a194c0a72c510f06da7ef20d0aa6019e89caf272494349a4b3a89ad0ed3f9c6a53ef1ed768f55d9edc6b40ceaef735080da5fb7fb5832df18b749b8b3fa84

data/CHANGELOG.md

@@ -2,12 +2,33 @@
 
 All notable changes to this project will be documented in this file. This project follows [Semantic Versioning](https://semver.org/).
 
+## 4.3.0 — 2026-10-24
+### Added
+
+- `Client#serp` gains `pages:`, `location:`, `uule:`, `lat:`, `lon:` and `radius:`. Every search runs from one fixed location (`location`, `lat` + `lon`, or `uule`, otherwise the capital of the `gl` country), so repeats and later pages are consistent; `gl` defaults to the location's country. `pages` (1–5) fetches consecutive pages as one de-duplicated list, billed per results page returned (see https://webscraping.ai/docs#serp). Raises `ArgumentError` when `pages` or `radius` is not an Integer >= 1, or when `lat`/`lon` is not a finite number. `lat`/`lon` are sent as plain decimals rounded to 7 places (never exponent notation). Ranges and conflicting location options are left to the server's 400 (not charged). New response keys are documented in the README (`search_parameters.location_used`, `organic_results[].page`, `pagination.pages_returned`, ...).
+- `Client#serp_locations(q = nil, limit: nil)` for the new `GET /serp/locations` endpoint — returns an Array of matching locations (`id`, `canonical_name`, `country_code`, `target_type`, `reach`, `gps`, ...), most populous first; pass `canonical_name` or `id` as `serp`'s `location:`. Free, and the endpoint needs no API key. Raises `ArgumentError` when `limit` is not an Integer >= 1.
+- `bin/smoke.rb` checks a two-page SERP from Austin and `serp_locations("austin", limit: 3)`.
+
+### Changed
+
+- Docs: stop stating credit prices (they're set server-side and change); link to https://webscraping.ai/docs pricing instead.
+
+## 4.2.0 — 2026-09-25
+### Added
+
+- `Client#data(url, country: nil, transcript: nil, transcript_language: nil, **params)` for the `GET /data` endpoint — structured JSON for a page on a supported site (e.g. YouTube, TikTok, X, LinkedIn, Instagram, Reddit) as a Hash with `request_parameters`, `parse_status` and `data`. Priced per site (see https://webscraping.ai/docs#data). The client does not check the URL's site: new sites are added server-side, and an unsupported URL or page type returns a 400 that is not charged (`BadRequestError`); its message lists what is supported. Raises `ArgumentError` when `url` is blank or not a String. Extra keyword arguments are sent as-is as query params (String, Integer, Float or boolean; Floats are sent as plain decimals). `api_key`, `url`, `country`, `transcript` and `transcript_language` as extra params raise `ArgumentError` (use the named options).
+- `bin/smoke.rb` checks `/data` on a YouTube video and that `https://example.com/` gets the server's 400 "Unsupported URL" error.
+
+### Fixed
+
+- `TimeoutError` and `ConnectionError` redact `api_key=...` from the transport error's message and no longer carry the Faraday exception as `cause`, since it may include the request URL.
+
 ## 4.1.0 — 2026-09-25
 
 ### Added
 
-- `Client#serp(q:, engine: nil, gl: nil, hl: nil, page: nil)` for the new `GET /serp` endpoint — parsed Google search results (organic results, related searches, pagination) as a Hash. Flat 15 credits per search. Raises `ArgumentError` when `q` is blank or not a String, or when `page` is not an Integer >= 1 (the server also rejects it with a 400, not billed; checking client-side saves the round trip). Pages are 1–100: the server rejects a `page` above 100 with a 400.
-- `bin/smoke.rb` live smoke script: asserts result shapes (not just the absence of exceptions), runs page tools with `js: false` on datacenter proxies (~32 credits per sweep), and redacts the API key from failure output.
+- `Client#serp(q:, engine: nil, gl: nil, hl: nil, page: nil)` for the new `GET /serp` endpoint — parsed Google search results (organic results, related searches, pagination) as a Hash. Priced per search (see https://webscraping.ai/docs#serp). Raises `ArgumentError` when `q` is blank or not a String, or when `page` is not an Integer >= 1 (the server also rejects it with a 400, not billed; checking client-side saves the round trip). Pages are 1–100: the server rejects a `page` above 100 with a 400.
+- `bin/smoke.rb` live smoke script: asserts result shapes (not just the absence of exceptions), runs page tools with `js: false` on datacenter proxies to keep the sweep cheap, and redacts the API key from failure output.
 
 ### Fixed
 

data/README.md

@@ -26,8 +26,8 @@ Requires Ruby 3.1+.
 
 ## Quick start
 
-[Sign up](https://webscraping.ai/auth/sign_up) to get an API key — the free
-trial includes 2,000 credits, no credit card required. Your key lives in the
+[Sign up](https://webscraping.ai/auth/sign_up) to get an API key — a free
+trial, no credit card required. Your key lives in the
 [dashboard](https://webscraping.ai/dashboard).
 
 ```ruby
@@ -64,6 +64,16 @@ data = client.fields(
 results = client.serp(q: "coffee machines", gl: "us", hl: "en", page: 1)
 results["organic_results"].first["link"]
 
+# Two pages of results searched from Austin, as one de-duplicated list
+results = client.serp(q: "coffee machines", location: "Austin, TX", pages: 2)
+
+# Location names for serp's location: (free)
+client.serp_locations("austin").first["canonical_name"] # => "Austin,Texas,United States"
+
+# Structured data for a page on a supported site (YouTube, TikTok, X, LinkedIn, Instagram, Reddit, ...)
+video = client.data("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
+video["data"]["title"]
+
 # Check your account quota
 info = client.account
 # => { "remaining_api_calls" => 200_000, "resets_at" => 1_617_073_667, "remaining_concurrency" => 100 }
@@ -128,15 +138,28 @@ Returns: `String` for HTML/text responses, `Hash`/`Array` for JSON responses.
 ### SERP (`#serp`)
 
 `#serp` is query-shaped rather than URL-shaped, so none of the page-fetch options above apply.
-It returns the parsed search results as a `Hash`. Flat 15 credits per search; failed searches are not charged.
+It returns the parsed search results as a `Hash`. Billed per results page returned (see [pricing](https://webscraping.ai/docs#serp)); failed searches are not charged.
+
+Every search runs from one fixed location: `location`, `lat` + `lon`, or `uule`, otherwise the capital of
+the `gl` country (for the few countries outside the location list, Google places the search by the exit IP),
+so repeats and later pages are consistent. `gl` defaults to the location's country.
+`pages` (1–5) fetches consecutive pages, starting at `page`, as one de-duplicated list, billed per page returned.
 
 | Option | Type | Default | Description |
 | --- | --- | --- | --- |
 | `q` | `String` | — | Search query (required; blank or non-String raises `ArgumentError`) |
 | `engine` | `String` | `"google"` | Search engine; currently only `google` |
-| `gl` | `String` | `"us"` | Two-letter country code for the search |
+| `gl` | `String` | `"us"` | Two-letter country code for the search. Defaults to the country of `location`, `lat`/`lon` or `uule`, otherwise `us` |
 | `hl` | `String` | `"en"` | Two-letter language code for the results |
 | `page` | `Integer` | `1` | Results page number (10 results per page). Must be an `Integer` >= 1, otherwise `ArgumentError`; the server rejects values above 100 with a 400 (not billed) |
+| `pages` | `Integer` | `1` | Number of consecutive results pages to fetch in one call, starting at `page`, from 1 to 5. Duplicates across pages are removed, `position` counts through the combined list and each result gets a `page` key. Must be an `Integer` >= 1, otherwise `ArgumentError` |
+| `location` | `String` | — | Where the search runs from: a name such as `"Austin,Texas,United States"` or `"Austin, TX"`, or an `id` from `#serp_locations`. Large cities also match local names, e.g. `"Москва"`. Sent as-is; an unknown name returns a 400 (not billed). Cannot be combined with `uule`, `lat` or `lon` |
+| `uule` | `String` | — | Google-encoded location (`w+CAIQICI…` or `a+…`). Cannot be combined with `location`, `lat` or `lon` |
+| `lat` | `Float` | — | Latitude of the search origin in decimal degrees (-90 to 90). Requires `lon`. Must be a finite number, otherwise `ArgumentError`; sent as a plain decimal rounded to 7 places |
+| `lon` | `Float` | — | Longitude of the search origin in decimal degrees (-180 to 180). Requires `lat`. Same rules as `lat` |
+| `radius` | `Integer` | — | Radius in meters around `location` or `lat`/`lon`, from 1 to 1000. Must be an `Integer` >= 1, otherwise `ArgumentError` |
+
+Ranges and which location options can be combined are checked by the server (a 400, not billed).
 
 ```ruby
 results = client.serp(q: "coffee machines", gl: "gb", page: 2)
@@ -145,13 +168,80 @@ results["organic_results"].each do |r|
   puts "#{r["position"]}. #{r["title"]} — #{r["link"]}"
 end
 results["pagination"] # => { "current" => 2, "next" => 3 }
+
+results = client.serp(q: "coffee machines", location: "Austin, TX", pages: 2)
+results["search_parameters"]["location_used"] # => "Austin,Texas,United States"
+results["organic_results"].last["page"]       # => 2 (position counts through both pages)
+results["pagination"]["pages_returned"] # => 2
 ```
 
-Response keys: `search_parameters` (`engine`, `q`, `gl`, `hl`, `page`), `search_information`
+Response keys: `search_parameters` (`engine`, `q`, `gl`, `hl`, `page`; `location_used`, `lat` and `lon`, usually
+present, which are the `gl` country's capital when no location is given; `location_requested` and `radius` when given;
+`pages` only above 1), `search_information`
 (`query_displayed`, `organic_results_state`, optional `showing_results_for` and `total_results`),
-`organic_results` (`position` — 1-based within the page — `title`, `link`, `domain`, `displayed_link`,
-optional `snippet` and `date`), optional `related_searches` (`query`), and `pagination` (`current`, optional `next`).
-Optional keys are absent when Google does not show them.
+`organic_results` (`position` — 1-based within the page, or through the combined list with `pages` above 1 —
+`title`, `link`, `domain`, `displayed_link`, optional `snippet` and `date`, and `page` with `pages` above 1),
+optional `related_searches` (`query`), and `pagination` (`current`, optional `next`, and `pages_returned` with
+`pages` above 1). Optional keys are absent when Google does not show them.
+
+### SERP locations (`#serp_locations`)
+
+`#serp_locations(q = nil, limit: nil)` looks up names for `#serp`'s `location:` option. It is free and the
+endpoint needs no API key (the client still sends yours). It returns an `Array` of up to 10 location `Hash`es,
+most populous first. `q` is part of an English location name (without it, the most populous locations are
+returned); `limit` must be an `Integer` >= 1 (the server allows up to 10).
+
+```ruby
+locations = client.serp_locations("austin", limit: 3)
+locations.first
+# => { "id" => "1026201", "google_id" => 1026201, "google_parent_id" => 21176, "name" => "Austin",
+#      "canonical_name" => "Austin,Texas,United States", "country_code" => "US", "target_type" => "City",
+#      "reach" => 974447, "gps" => [-97.7431, 30.2672] }
+
+client.serp(q: "coffee machines", location: locations.first["canonical_name"]) # or its "id"
+```
+
+`gps` is `[longitude, latitude]`, `target_type` is an open set of strings (e.g. `City`, `State`, `Country`),
+and `google_parent_id` is `nil` for countries.
+
+### Structured data (`#data`)
+
+`#data(url, country: nil, transcript: nil, transcript_language: nil, **params)` returns structured JSON
+for a public page on a supported site as a `Hash`. Pass the page's normal URL; the site (`provider`) and
+page kind (`type`) are detected from it. Priced per site (see [pricing](https://webscraping.ai/docs#data)),
+including pages that parse empty (`parse_status` `"parse_failed"`) or no longer exist (`"not_found"`);
+unsupported URLs and failed fetches are not charged.
+None of the page-fetch options above apply.
+
+Supported sites today include, for example, YouTube (video/channel/playlist), TikTok (video/profile),
+X/Twitter (tweet/profile), LinkedIn (company/job/profile), Instagram (post/reel/profile) and Reddit
+(post/subreddit/user). **More sites are added server-side**, and they work with this gem without an
+upgrade: the client never checks the URL's site. An unsupported URL or page type returns a 400 that is
+not charged (`WebScrapingAI::BadRequestError`). Its message lists what is supported.
+
+| Option | Type | Default | Description |
+| --- | --- | --- | --- |
+| `url` | `String` | — | Page URL (required, positional; blank or non-String raises `ArgumentError`) |
+| `country` | `String` | `"us"` | Two-letter country code of the proxy used to fetch the page, `us` by default |
+| `transcript` | `Boolean` | `false` | YouTube videos only. Also fetch the video's transcript into `data.transcript`. It's null when no matching captions are available. If the transcript fetch itself fails, the whole request fails with a 500 and is not charged |
+| `transcript_language` | `String` | — | Caption language to pick, e.g. `en` or `de`. Without it, English is preferred, then the first available track. If the video has no captions in that language, `data.transcript` is null |
+| `**params` | `String`, `Integer`, `Float`, boolean | — | Extra query params sent as-is, for provider-specific params added later (`nil` omits one). `api_key`, `url`, `country`, `transcript` and `transcript_language` raise `ArgumentError`; use the named options for the last three |
+
+```ruby
+result = client.data("https://www.youtube.com/watch?v=dQw4w9WgXcQ", transcript: true)
+result["request_parameters"] # => { "url" => "...", "provider" => "youtube", "type" => "video" }
+result["parse_status"]       # => "ok" (or "parse_failed" / "not_found")
+result["data"]["title"]      # shape depends on provider and type; may be nil
+
+begin
+  client.data("https://example.com/")
+rescue WebScrapingAI::BadRequestError => e
+  e.message # => "Unsupported URL for /data. Supported sites: youtube, tiktok, ..."
+end
+```
+
+`provider`, `type` and `parse_status` are open sets of strings, and `data` is the decoded JSON as-is
+(no per-site classes), so new sites and fields show up without a gem release.
 
 ## Error handling
 
@@ -191,7 +281,7 @@ bundle exec rubocop
 
 ## Smoke testing
 
-`bin/smoke.rb` hits every endpoint once against the live API, loading the gem from `lib/` so it tests the working tree. It is not part of the spec suite and costs ~32 credits per run: the four page calls run with `js: false` and `proxy: "datacenter"` (1 credit each), `question` and `fields` cost 6 each, and the SERP call is 15. Each case checks the result shape as well as exceptions (e.g. SERP must return organic results for the right query, `selected_multiple` must match something), and failure messages redact the API key.
+`bin/smoke.rb` hits every endpoint once against the live API, loading the gem from `lib/` so it tests the working tree. It is not part of the spec suite and uses real credits on every run (see [pricing](https://webscraping.ai/docs)): the four page calls run with `js: false` and `proxy: "datacenter"` to keep them cheap, there are two AI calls (`question`, `fields`), two SERP calls (one plain, one two-page search from Austin, billed per page returned) and one `/data` call (a YouTube video). `serp_locations` is free. A second `/data` call on `https://example.com/` must come back as the server's free 400, proving there is no client-side site filter. Each case checks the result shape as well as exceptions (e.g. SERP must return organic results for the right query, the Austin SERP must report an Austin `location_used`, `serp_locations` must find Austin, `/data` must parse a title, `selected_multiple` must match something), and failure messages redact the API key.
 
 ```bash
 WEBSCRAPING_AI_API_KEY=... bundle exec rake smoke

data/lib/webscraping_ai/client.rb

@@ -69,15 +69,71 @@ module WebScrapingAI
 
     # GET /serp — returns parsed search engine results for `q` as a Hash
     # (search_parameters, search_information, organic_results, related_searches, pagination).
-    # Query-shaped: none of the page-fetch options apply. Flat 15 credits per search.
+    # Query-shaped: none of the page-fetch options apply. Billed per results page returned
+    # (see https://webscraping.ai/docs#serp); failed searches are not charged.
     # `q` must be a non-blank String (sent as-is, untrimmed). `page`, when given, must be an
     # Integer >= 1; the server rejects values above 100 with a 400 (not billed).
-    def serp(q:, engine: nil, gl: nil, hl: nil, page: nil)
+    #
+    # Every search runs from one fixed location: `location` (a name such as "Austin, TX" or
+    # "Austin,Texas,United States", or an id from #serp_locations), `lat` + `lon`, or `uule`, otherwise
+    # the capital of the `gl` country, so repeats and later pages are consistent. `gl` defaults to the
+    # location's country. `pages` (1-5) fetches consecutive pages starting at `page` as one de-duplicated
+    # list: `position` counts through the combined list and each result gets a `page` key.
+    #
+    # `pages` and `radius` (meters, 1-1000), when given, must be Integers >= 1; `lat` and `lon` must be
+    # finite numbers and are sent as plain decimals. Ranges and which location params can be combined are
+    # checked by the server (a 400, not billed).
+    def serp(q:, engine: nil, gl: nil, hl: nil, page: nil, pages: nil, location: nil, uule: nil, lat: nil,
+             lon: nil, radius: nil)
       raise ArgumentError, "q is required" if q.nil? || (q.is_a?(String) && q.strip.empty?)
       raise ArgumentError, "q must be a String" unless q.is_a?(String)
-      raise ArgumentError, "page must be an Integer >= 1" unless page.nil? || (page.is_a?(Integer) && page >= 1)
 
-      get("/serp", q: q, engine: engine, gl: gl, hl: hl, page: page)
+      check_positive_integer(:page, page)
+      check_positive_integer(:pages, pages)
+      check_positive_integer(:radius, radius)
+
+      get("/serp", q: q, engine: engine, gl: gl, hl: hl, page: page, pages: pages, location: location,
+                   uule: uule, lat: coordinate(:lat, lat), lon: coordinate(:lon, lon), radius: radius)
+    end
+
+    # GET /serp/locations — returns an Array of location Hashes matching `q` (part of an English
+    # location name), most populous first: id, google_id, google_parent_id (nil for countries), name,
+    # canonical_name, country_code, target_type, reach and gps ([longitude, latitude]). Pass
+    # `canonical_name` (or `id`) as #serp's `location:`. Without `q`, the most populous locations are
+    # returned. `limit`, when given, must be an Integer >= 1; the server allows up to 10. Free.
+    def serp_locations(q = nil, limit: nil)
+      check_positive_integer(:limit, limit)
+
+      get("/serp/locations", q: q, limit: limit)
+    end
+
+    # GET /data — returns structured JSON for a page on a supported site as a Hash
+    # (request_parameters: url/provider/type, parse_status, data). Priced per site
+    # (see https://webscraping.ai/docs#data); unsupported URLs and failed fetches are not charged.
+    #
+    # The client never checks which site `url` belongs to: supported sites (e.g. YouTube, TikTok,
+    # X/Twitter, LinkedIn, Instagram, Reddit) are added server-side. An unsupported URL or page type
+    # returns a 400 that is not charged (BadRequestError); its message lists what is supported.
+    # `provider`, `type` and `parse_status` are open sets of strings; `data` may be nil.
+    #
+    # `country`: two-letter country code of the proxy used to fetch the page, `us` by default.
+    # `transcript`: YouTube videos only. Also fetch the video's transcript into `data.transcript`.
+    # It's null when no matching captions are available. If the transcript fetch itself fails, the
+    # whole request fails with a 500 and is not charged.
+    # `transcript_language`: caption language to pick, e.g. `en` or `de`. Without it, English is
+    # preferred, then the first available track. If the video has no captions in that language,
+    # `data.transcript` is null.
+    #
+    # Any other keyword arguments are sent as-is as extra query params (for provider-specific params
+    # added later). Values must be String, Integer, Float or boolean (nil omits the param); the names
+    # `api_key`, `url`, `country`, `transcript` and `transcript_language` raise ArgumentError.
+    # None of the page-fetch options apply.
+    def data(url, country: nil, transcript: nil, transcript_language: nil, **params)
+      raise ArgumentError, "url is required" if url.nil? || (url.is_a?(String) && url.strip.empty?)
+      raise ArgumentError, "url must be a String" unless url.is_a?(String)
+
+      named = { country: country, transcript: transcript, transcript_language: transcript_language }.compact
+      get("/data", url: url, **data_extra_params(params), **named)
     end
 
     def inspect
@@ -91,6 +147,73 @@ module WebScrapingAI
 
     private
 
+    DATA_RESERVED_PARAMS = %w[api_key url].freeze
+    DATA_TYPED_PARAMS = %w[country transcript transcript_language].freeze
+    private_constant :DATA_RESERVED_PARAMS, :DATA_TYPED_PARAMS
+
+    # Extra /data query params are passed through untouched (same encoder, so `&`/`=` are escaped),
+    # except that they can't override the credentials or the target URL, and can't repeat a typed
+    # param (a string key like "country" would otherwise collide with `country:`).
+    def data_extra_params(params)
+      params.each_with_object({}) do |(key, value), extra|
+        name = key.to_s
+        raise ArgumentError, "#{name} can't be passed as an extra /data param" if DATA_RESERVED_PARAMS.include?(name)
+        if DATA_TYPED_PARAMS.include?(name)
+          raise ArgumentError, "#{name} can't be passed as an extra /data param; use the #{name}: option"
+        end
+
+        extra[name.to_sym] = data_extra_value(name, value)
+      end
+    end
+
+    def data_extra_value(name, value)
+      case value
+      when nil, String, Integer, true, false then value
+      when Float
+        raise ArgumentError, "extra /data param #{name} must be a finite number" unless value.finite?
+
+        plain_float(value)
+      else
+        raise ArgumentError, "extra /data param #{name} must be a String, Integer, Float or boolean"
+      end
+    end
+
+    def check_positive_integer(name, value)
+      return if value.nil? || (value.is_a?(Integer) && value >= 1)
+
+      raise ArgumentError, "#{name} must be an Integer >= 1"
+    end
+
+    # The server only accepts plain decimals (`-?\d+(\.\d+)?`) for lat/lon, so never send Ruby's
+    # exponent notation (1.0e-07): round to 7 fractional digits and trim trailing zeros.
+    def coordinate(name, value)
+      return nil if value.nil?
+      raise ArgumentError, "#{name} must be a finite number" unless value.is_a?(Numeric) && value.real? && value.finite?
+
+      text = format("%.7f", value).sub(/0+\z/, "").delete_suffix(".")
+      # BigDecimal("1e400") is finite but formats as "Inf".
+      raise ArgumentError, "#{name} must be a finite number" unless text.match?(/\A-?\d+(\.\d+)?\z/)
+
+      text == "-0" ? "0" : text
+    end
+
+    # Float#to_s switches to exponent notation (1.0e+20, 1.5e-07); send plain decimals instead.
+    def plain_float(value)
+      text = value.to_s
+      return text unless text.include?("e")
+
+      mantissa, exponent = text.split("e")
+      fraction_digits = mantissa.split(".")[1].to_s.sub(/0+\z/, "").length
+      format("%.#{[fraction_digits - exponent.to_i, 0].max}f", value)
+    end
+
+    def redact(text)
+      key = configuration.api_key.to_s
+      text = text.to_s
+      text = text.gsub(key, "[FILTERED]") unless key.empty?
+      text.gsub(/api_key=[^&\s"']*/, "api_key=[FILTERED]")
+    end
+
     def connection
       @connection ||= Faraday.new(url: configuration.base_url) do |conn|
         conn.options.timeout = configuration.timeout
@@ -102,15 +225,17 @@ module WebScrapingAI
       end
     end
 
+    # Transport errors are re-raised with `cause: nil` and a redacted message: the Faraday error (and
+    # anything it wraps) may carry the request URL, which contains the API key.
     def get(path, **params)
       response = connection.get(path) do |req|
         req.params = params.merge(api_key: configuration.api_key)
       end
       handle_response(response)
     rescue Faraday::TimeoutError => e
-      raise TimeoutError, e.message
+      raise TimeoutError.new(redact(e.message)), cause: nil
     rescue Faraday::ConnectionFailed => e
-      raise ConnectionError, e.message
+      raise ConnectionError.new(redact(e.message)), cause: nil
     end
 
     def handle_response(response)

data/lib/webscraping_ai/version.rb

@@ -1,3 +1,3 @@
 module WebScrapingAI
-  VERSION = "4.1.0".freeze
+  VERSION = "4.3.0".freeze
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: webscraping_ai
 version: !ruby/object:Gem::Version
-  version: 4.1.0
+  version: 4.3.0
 platform: ruby
 authors:
 - WebScraping.AI