webscraping_ai 2.0.2 → 3.1.3

data/docs/SelectedHTMLApi.md

@@ -2,13 +2,10 @@
 
 All URIs are relative to *https://api.webscraping.ai*
 
-Method | HTTP request | Description
-------------- | ------------- | -------------
-[**get_selected**](SelectedHTMLApi.md#get_selected) | **GET** /selected | HTML of a selected page area by URL and CSS selector
-[**get_selected_multiple**](SelectedHTMLApi.md#get_selected_multiple) | **GET** /selected-multiple | HTML of multiple page areas by URL and CSS selectors
-[**post_selected**](SelectedHTMLApi.md#post_selected) | **POST** /selected | HTML of a selected page areas by URL and CSS selector, with POST request to the target page
-[**post_selected_multiple**](SelectedHTMLApi.md#post_selected_multiple) | **POST** /selected-multiple | HTML of multiple page areas by URL and CSS selectors, with POST request to the target page
-
+| Method | HTTP request | Description |
+| ------ | ------------ | ----------- |
+| [**get_selected**](SelectedHTMLApi.md#get_selected) | **GET** /selected | HTML of a selected page area by URL and CSS selector |
+| [**get_selected_multiple**](SelectedHTMLApi.md#get_selected_multiple) | **GET** /selected-multiple | HTML of multiple page areas by URL and CSS selectors |
 
 
 ## get_selected
@@ -17,121 +14,84 @@ Method | HTTP request | Description
 
 HTML of a selected page area by URL and CSS selector
 
-Returns just HTML on success, JSON on error
+Returns HTML of a selected page area by URL and CSS selector. Useful if you don't want to do the HTML parsing on your side.
 
-### Example
+### Examples
 
 ```ruby
-# load the gem
+require 'time'
 require 'webscraping_ai'
 # setup authorization
 WebScrapingAI.configure do |config|
   # Configure API key authorization: api_key
   config.api_key['api_key'] = 'YOUR API KEY'
   # Uncomment the following line to set a prefix for the API key, e.g. 'Bearer' (defaults to nil)
-  #config.api_key_prefix['api_key'] = 'Bearer'
+  # config.api_key_prefix['api_key'] = 'Bearer'
 end
 
 api_instance = WebScrapingAI::SelectedHTMLApi.new
-url = 'https://example.com' # String | URL of the target page
+url = 'https://example.com' # String | URL of the target page.
 opts = {
   selector: 'h1', # String | CSS selector (null by default, returns whole page HTML)
-  headers: {'key' => '{\"Cookie\":\"session=some_id\"}'}, # Hash<String, String> | HTTP headers to pass to the target page. Can be specified either via a nested query parameter (...&headers[One]=value1&headers=[Another]=value2) or as a JSON encoded object (...&headers={\"One\": \"value1\", \"Another\": \"value2\"})
-  timeout: 5000, # Integer | Maximum processing time in ms. Increase it in case of timeout errors (5000 by default, maximum is 30000)
-  js: true, # Boolean | Execute on-page JavaScript using a headless browser (true by default), costs 2 requests
-  proxy: 'datacenter' # String | Type of proxy, use residential proxies if your site restricts traffic from datacenters (datacenter by default)
+  headers: { key: 3.56}, # Hash<String, String> | HTTP headers to pass to the target page. Can be specified either via a nested query parameter (...&headers[One]=value1&headers=[Another]=value2) or as a JSON encoded object (...&headers={\"One\": \"value1\", \"Another\": \"value2\"}).
+  timeout: 10000, # Integer | Maximum web page retrieval time in ms. Increase it in case of timeout errors (10000 by default, maximum is 30000).
+  js: true, # Boolean | Execute on-page JavaScript using a headless browser (true by default).
+  js_timeout: 2000, # Integer | Maximum JavaScript rendering time in ms. Increase it in case if you see a loading indicator instead of data on the target page.
+  proxy: 'datacenter', # String | Type of proxy, use residential proxies if your site restricts traffic from datacenters (datacenter by default). Note that residential proxy requests are more expensive than datacenter, see the pricing page for details.
+  country: 'us', # String | Country of the proxy to use (US by default). Only available on Startup and Custom plans.
+  device: 'desktop', # String | Type of device emulation.
+  error_on_404: false, # Boolean | Return error on 404 HTTP status on the target page (false by default).
+  error_on_redirect: false, # Boolean | Return error on redirect on the target page (false by default).
+  js_script: 'document.querySelector('button').click();' # String | Custom JavaScript code to execute on the target page.
 }
 
 begin
-  #HTML of a selected page area by URL and CSS selector
+  # HTML of a selected page area by URL and CSS selector
   result = api_instance.get_selected(url, opts)
   p result
 rescue WebScrapingAI::ApiError => e
-  puts "Exception when calling SelectedHTMLApi->get_selected: #{e}"
+  puts "Error when calling SelectedHTMLApi->get_selected: #{e}"
 end
 ```
 
-### Parameters
-
+#### Using the get_selected_with_http_info variant
 
-Name | Type | Description  | Notes
-------------- | ------------- | ------------- | -------------
- **url** | **String**| URL of the target page | 
- **selector** | **String**| CSS selector (null by default, returns whole page HTML) | [optional] 
- **headers** | [**Hash&lt;String, String&gt;**](String.md)| HTTP headers to pass to the target page. Can be specified either via a nested query parameter (...&amp;headers[One]&#x3D;value1&amp;headers&#x3D;[Another]&#x3D;value2) or as a JSON encoded object (...&amp;headers&#x3D;{\&quot;One\&quot;: \&quot;value1\&quot;, \&quot;Another\&quot;: \&quot;value2\&quot;}) | [optional] 
- **timeout** | **Integer**| Maximum processing time in ms. Increase it in case of timeout errors (5000 by default, maximum is 30000) | [optional] [default to 5000]
- **js** | **Boolean**| Execute on-page JavaScript using a headless browser (true by default), costs 2 requests | [optional] [default to true]
- **proxy** | **String**| Type of proxy, use residential proxies if your site restricts traffic from datacenters (datacenter by default) | [optional] [default to &#39;datacenter&#39;]
+This returns an Array which contains the response data, status code and headers.
 
-### Return type
-
-**String**
-
-### Authorization
-
-[api_key](../README.md#api_key)
-
-### HTTP request headers
-
-- **Content-Type**: Not defined
-- **Accept**: application/json, text/html
-
-
-## get_selected_multiple
-
-> Array&lt;String&gt; get_selected_multiple(url, opts)
-
-HTML of multiple page areas by URL and CSS selectors
-
-Always returns JSON
-
-### Example
+> <Array(String, Integer, Hash)> get_selected_with_http_info(url, opts)
 
 ```ruby
-# load the gem
-require 'webscraping_ai'
-# setup authorization
-WebScrapingAI.configure do |config|
-  # Configure API key authorization: api_key
-  config.api_key['api_key'] = 'YOUR API KEY'
-  # Uncomment the following line to set a prefix for the API key, e.g. 'Bearer' (defaults to nil)
-  #config.api_key_prefix['api_key'] = 'Bearer'
-end
-
-api_instance = WebScrapingAI::SelectedHTMLApi.new
-url = 'https://example.com' # String | URL of the target page
-opts = {
-  selectors: ['[\"h1\"]'], # Array<String> | Multiple CSS selectors (null by default, returns whole page HTML)
-  headers: {'key' => '{\"Cookie\":\"session=some_id\"}'}, # Hash<String, String> | HTTP headers to pass to the target page. Can be specified either via a nested query parameter (...&headers[One]=value1&headers=[Another]=value2) or as a JSON encoded object (...&headers={\"One\": \"value1\", \"Another\": \"value2\"})
-  timeout: 5000, # Integer | Maximum processing time in ms. Increase it in case of timeout errors (5000 by default, maximum is 30000)
-  js: true, # Boolean | Execute on-page JavaScript using a headless browser (true by default), costs 2 requests
-  proxy: 'datacenter' # String | Type of proxy, use residential proxies if your site restricts traffic from datacenters (datacenter by default)
-}
-
 begin
-  #HTML of multiple page areas by URL and CSS selectors
-  result = api_instance.get_selected_multiple(url, opts)
-  p result
+  # HTML of a selected page area by URL and CSS selector
+  data, status_code, headers = api_instance.get_selected_with_http_info(url, opts)
+  p status_code # => 2xx
+  p headers # => { ... }
+  p data # => String
 rescue WebScrapingAI::ApiError => e
-  puts "Exception when calling SelectedHTMLApi->get_selected_multiple: #{e}"
+  puts "Error when calling SelectedHTMLApi->get_selected_with_http_info: #{e}"
 end
 ```
 
 ### Parameters
 
-
-Name | Type | Description  | Notes
-------------- | ------------- | ------------- | -------------
- **url** | **String**| URL of the target page | 
- **selectors** | [**Array&lt;String&gt;**](String.md)| Multiple CSS selectors (null by default, returns whole page HTML) | [optional] 
- **headers** | [**Hash&lt;String, String&gt;**](String.md)| HTTP headers to pass to the target page. Can be specified either via a nested query parameter (...&amp;headers[One]&#x3D;value1&amp;headers&#x3D;[Another]&#x3D;value2) or as a JSON encoded object (...&amp;headers&#x3D;{\&quot;One\&quot;: \&quot;value1\&quot;, \&quot;Another\&quot;: \&quot;value2\&quot;}) | [optional] 
- **timeout** | **Integer**| Maximum processing time in ms. Increase it in case of timeout errors (5000 by default, maximum is 30000) | [optional] [default to 5000]
- **js** | **Boolean**| Execute on-page JavaScript using a headless browser (true by default), costs 2 requests | [optional] [default to true]
- **proxy** | **String**| Type of proxy, use residential proxies if your site restricts traffic from datacenters (datacenter by default) | [optional] [default to &#39;datacenter&#39;]
+| Name | Type | Description | Notes |
+| ---- | ---- | ----------- | ----- |
+| **url** | **String** | URL of the target page. |  |
+| **selector** | **String** | CSS selector (null by default, returns whole page HTML) | [optional] |
+| **headers** | [**Hash&lt;String, String&gt;**](String.md) | HTTP headers to pass to the target page. Can be specified either via a nested query parameter (...&amp;headers[One]&#x3D;value1&amp;headers&#x3D;[Another]&#x3D;value2) or as a JSON encoded object (...&amp;headers&#x3D;{\&quot;One\&quot;: \&quot;value1\&quot;, \&quot;Another\&quot;: \&quot;value2\&quot;}). | [optional] |
+| **timeout** | **Integer** | Maximum web page retrieval time in ms. Increase it in case of timeout errors (10000 by default, maximum is 30000). | [optional][default to 10000] |
+| **js** | **Boolean** | Execute on-page JavaScript using a headless browser (true by default). | [optional][default to true] |
+| **js_timeout** | **Integer** | Maximum JavaScript rendering time in ms. Increase it in case if you see a loading indicator instead of data on the target page. | [optional][default to 2000] |
+| **proxy** | **String** | Type of proxy, use residential proxies if your site restricts traffic from datacenters (datacenter by default). Note that residential proxy requests are more expensive than datacenter, see the pricing page for details. | [optional][default to &#39;datacenter&#39;] |
+| **country** | **String** | Country of the proxy to use (US by default). Only available on Startup and Custom plans. | [optional][default to &#39;us&#39;] |
+| **device** | **String** | Type of device emulation. | [optional][default to &#39;desktop&#39;] |
+| **error_on_404** | **Boolean** | Return error on 404 HTTP status on the target page (false by default). | [optional][default to false] |
+| **error_on_redirect** | **Boolean** | Return error on redirect on the target page (false by default). | [optional][default to false] |
+| **js_script** | **String** | Custom JavaScript code to execute on the target page. | [optional] |
 
 ### Return type
 
-**Array&lt;String&gt;**
+**String**
 
 ### Authorization
 
@@ -140,130 +100,89 @@ Name | Type | Description  | Notes
 ### HTTP request headers
 
 - **Content-Type**: Not defined
-- **Accept**: application/json
+- **Accept**: application/json, text/html
 
 
-## post_selected
+## get_selected_multiple
 
-> String post_selected(url, opts)
+> Array&lt;String&gt; get_selected_multiple(url, opts)
 
-HTML of a selected page areas by URL and CSS selector, with POST request to the target page
+HTML of multiple page areas by URL and CSS selectors
 
-Returns just HTML on success, JSON on error. Request body will be passed to the target page.
+Returns HTML of multiple page areas by URL and CSS selectors. Useful if you don't want to do the HTML parsing on your side.
 
-### Example
+### Examples
 
 ```ruby
-# load the gem
+require 'time'
 require 'webscraping_ai'
 # setup authorization
 WebScrapingAI.configure do |config|
   # Configure API key authorization: api_key
   config.api_key['api_key'] = 'YOUR API KEY'
   # Uncomment the following line to set a prefix for the API key, e.g. 'Bearer' (defaults to nil)
-  #config.api_key_prefix['api_key'] = 'Bearer'
+  # config.api_key_prefix['api_key'] = 'Bearer'
 end
 
 api_instance = WebScrapingAI::SelectedHTMLApi.new
-url = 'https://httpbin.org/post' # String | URL of the target page
+url = 'https://example.com' # String | URL of the target page.
 opts = {
-  selector: 'h1', # String | CSS selector (null by default, returns whole page HTML)
-  headers: {'key' => '{\"Cookie\":\"session=some_id\"}'}, # Hash<String, String> | HTTP headers to pass to the target page. Can be specified either via a nested query parameter (...&headers[One]=value1&headers=[Another]=value2) or as a JSON encoded object (...&headers={\"One\": \"value1\", \"Another\": \"value2\"})
-  timeout: 5000, # Integer | Maximum processing time in ms. Increase it in case of timeout errors (5000 by default, maximum is 30000)
-  js: true, # Boolean | Execute on-page JavaScript using a headless browser (true by default), costs 2 requests
-  proxy: 'datacenter', # String | Type of proxy, use residential proxies if your site restricts traffic from datacenters (datacenter by default)
-  request_body: nil # Hash<String, Object> | Request body to pass to the target page
+  selectors: ['inner_example'], # Array<String> | Multiple CSS selectors (null by default, returns whole page HTML)
+  headers: { key: 3.56}, # Hash<String, String> | HTTP headers to pass to the target page. Can be specified either via a nested query parameter (...&headers[One]=value1&headers=[Another]=value2) or as a JSON encoded object (...&headers={\"One\": \"value1\", \"Another\": \"value2\"}).
+  timeout: 10000, # Integer | Maximum web page retrieval time in ms. Increase it in case of timeout errors (10000 by default, maximum is 30000).
+  js: true, # Boolean | Execute on-page JavaScript using a headless browser (true by default).
+  js_timeout: 2000, # Integer | Maximum JavaScript rendering time in ms. Increase it in case if you see a loading indicator instead of data on the target page.
+  proxy: 'datacenter', # String | Type of proxy, use residential proxies if your site restricts traffic from datacenters (datacenter by default). Note that residential proxy requests are more expensive than datacenter, see the pricing page for details.
+  country: 'us', # String | Country of the proxy to use (US by default). Only available on Startup and Custom plans.
+  device: 'desktop', # String | Type of device emulation.
+  error_on_404: false, # Boolean | Return error on 404 HTTP status on the target page (false by default).
+  error_on_redirect: false, # Boolean | Return error on redirect on the target page (false by default).
+  js_script: 'document.querySelector('button').click();' # String | Custom JavaScript code to execute on the target page.
 }
 
 begin
-  #HTML of a selected page areas by URL and CSS selector, with POST request to the target page
-  result = api_instance.post_selected(url, opts)
+  # HTML of multiple page areas by URL and CSS selectors
+  result = api_instance.get_selected_multiple(url, opts)
   p result
 rescue WebScrapingAI::ApiError => e
-  puts "Exception when calling SelectedHTMLApi->post_selected: #{e}"
+  puts "Error when calling SelectedHTMLApi->get_selected_multiple: #{e}"
 end
 ```
 
-### Parameters
-
+#### Using the get_selected_multiple_with_http_info variant
 
-Name | Type | Description  | Notes
-------------- | ------------- | ------------- | -------------
- **url** | **String**| URL of the target page | 
- **selector** | **String**| CSS selector (null by default, returns whole page HTML) | [optional] 
- **headers** | [**Hash&lt;String, String&gt;**](String.md)| HTTP headers to pass to the target page. Can be specified either via a nested query parameter (...&amp;headers[One]&#x3D;value1&amp;headers&#x3D;[Another]&#x3D;value2) or as a JSON encoded object (...&amp;headers&#x3D;{\&quot;One\&quot;: \&quot;value1\&quot;, \&quot;Another\&quot;: \&quot;value2\&quot;}) | [optional] 
- **timeout** | **Integer**| Maximum processing time in ms. Increase it in case of timeout errors (5000 by default, maximum is 30000) | [optional] [default to 5000]
- **js** | **Boolean**| Execute on-page JavaScript using a headless browser (true by default), costs 2 requests | [optional] [default to true]
- **proxy** | **String**| Type of proxy, use residential proxies if your site restricts traffic from datacenters (datacenter by default) | [optional] [default to &#39;datacenter&#39;]
- **request_body** | [**Hash&lt;String, Object&gt;**](Object.md)| Request body to pass to the target page | [optional] 
-
-### Return type
-
-**String**
-
-### Authorization
-
-[api_key](../README.md#api_key)
-
-### HTTP request headers
-
-- **Content-Type**: application/json, application/x-www-form-urlencoded, application/xml, text/plain
-- **Accept**: application/json, text/html
+This returns an Array which contains the response data, status code and headers.
 
-
-## post_selected_multiple
-
-> Array&lt;String&gt; post_selected_multiple(url, opts)
-
-HTML of multiple page areas by URL and CSS selectors, with POST request to the target page
-
-Always returns JSON. Request body will be passed to the target page.
-
-### Example
+> <Array(Array&lt;String&gt;, Integer, Hash)> get_selected_multiple_with_http_info(url, opts)
 
 ```ruby
-# load the gem
-require 'webscraping_ai'
-# setup authorization
-WebScrapingAI.configure do |config|
-  # Configure API key authorization: api_key
-  config.api_key['api_key'] = 'YOUR API KEY'
-  # Uncomment the following line to set a prefix for the API key, e.g. 'Bearer' (defaults to nil)
-  #config.api_key_prefix['api_key'] = 'Bearer'
-end
-
-api_instance = WebScrapingAI::SelectedHTMLApi.new
-url = 'https://httpbin.org/post' # String | URL of the target page
-opts = {
-  selectors: ['[\"h1\"]'], # Array<String> | Multiple CSS selectors (null by default, returns whole page HTML)
-  headers: {'key' => '{\"Cookie\":\"session=some_id\"}'}, # Hash<String, String> | HTTP headers to pass to the target page. Can be specified either via a nested query parameter (...&headers[One]=value1&headers=[Another]=value2) or as a JSON encoded object (...&headers={\"One\": \"value1\", \"Another\": \"value2\"})
-  timeout: 5000, # Integer | Maximum processing time in ms. Increase it in case of timeout errors (5000 by default, maximum is 30000)
-  js: true, # Boolean | Execute on-page JavaScript using a headless browser (true by default), costs 2 requests
-  proxy: 'datacenter', # String | Type of proxy, use residential proxies if your site restricts traffic from datacenters (datacenter by default)
-  request_body: nil # Hash<String, Object> | Request body to pass to the target page
-}
-
 begin
-  #HTML of multiple page areas by URL and CSS selectors, with POST request to the target page
-  result = api_instance.post_selected_multiple(url, opts)
-  p result
+  # HTML of multiple page areas by URL and CSS selectors
+  data, status_code, headers = api_instance.get_selected_multiple_with_http_info(url, opts)
+  p status_code # => 2xx
+  p headers # => { ... }
+  p data # => Array&lt;String&gt;
 rescue WebScrapingAI::ApiError => e
-  puts "Exception when calling SelectedHTMLApi->post_selected_multiple: #{e}"
+  puts "Error when calling SelectedHTMLApi->get_selected_multiple_with_http_info: #{e}"
 end
 ```
 
 ### Parameters
 
-
-Name | Type | Description  | Notes
-------------- | ------------- | ------------- | -------------
- **url** | **String**| URL of the target page | 
- **selectors** | [**Array&lt;String&gt;**](String.md)| Multiple CSS selectors (null by default, returns whole page HTML) | [optional] 
- **headers** | [**Hash&lt;String, String&gt;**](String.md)| HTTP headers to pass to the target page. Can be specified either via a nested query parameter (...&amp;headers[One]&#x3D;value1&amp;headers&#x3D;[Another]&#x3D;value2) or as a JSON encoded object (...&amp;headers&#x3D;{\&quot;One\&quot;: \&quot;value1\&quot;, \&quot;Another\&quot;: \&quot;value2\&quot;}) | [optional] 
- **timeout** | **Integer**| Maximum processing time in ms. Increase it in case of timeout errors (5000 by default, maximum is 30000) | [optional] [default to 5000]
- **js** | **Boolean**| Execute on-page JavaScript using a headless browser (true by default), costs 2 requests | [optional] [default to true]
- **proxy** | **String**| Type of proxy, use residential proxies if your site restricts traffic from datacenters (datacenter by default) | [optional] [default to &#39;datacenter&#39;]
- **request_body** | [**Hash&lt;String, Object&gt;**](Object.md)| Request body to pass to the target page | [optional] 
+| Name | Type | Description | Notes |
+| ---- | ---- | ----------- | ----- |
+| **url** | **String** | URL of the target page. |  |
+| **selectors** | [**Array&lt;String&gt;**](String.md) | Multiple CSS selectors (null by default, returns whole page HTML) | [optional] |
+| **headers** | [**Hash&lt;String, String&gt;**](String.md) | HTTP headers to pass to the target page. Can be specified either via a nested query parameter (...&amp;headers[One]&#x3D;value1&amp;headers&#x3D;[Another]&#x3D;value2) or as a JSON encoded object (...&amp;headers&#x3D;{\&quot;One\&quot;: \&quot;value1\&quot;, \&quot;Another\&quot;: \&quot;value2\&quot;}). | [optional] |
+| **timeout** | **Integer** | Maximum web page retrieval time in ms. Increase it in case of timeout errors (10000 by default, maximum is 30000). | [optional][default to 10000] |
+| **js** | **Boolean** | Execute on-page JavaScript using a headless browser (true by default). | [optional][default to true] |
+| **js_timeout** | **Integer** | Maximum JavaScript rendering time in ms. Increase it in case if you see a loading indicator instead of data on the target page. | [optional][default to 2000] |
+| **proxy** | **String** | Type of proxy, use residential proxies if your site restricts traffic from datacenters (datacenter by default). Note that residential proxy requests are more expensive than datacenter, see the pricing page for details. | [optional][default to &#39;datacenter&#39;] |
+| **country** | **String** | Country of the proxy to use (US by default). Only available on Startup and Custom plans. | [optional][default to &#39;us&#39;] |
+| **device** | **String** | Type of device emulation. | [optional][default to &#39;desktop&#39;] |
+| **error_on_404** | **Boolean** | Return error on 404 HTTP status on the target page (false by default). | [optional][default to false] |
+| **error_on_redirect** | **Boolean** | Return error on redirect on the target page (false by default). | [optional][default to false] |
+| **js_script** | **String** | Custom JavaScript code to execute on the target page. | [optional] |
 
 ### Return type
 
@@ -275,6 +194,6 @@ Name | Type | Description  | Notes
 
 ### HTTP request headers
 
-- **Content-Type**: application/json, application/x-www-form-urlencoded, application/xml, text/plain
+- **Content-Type**: Not defined
 - **Accept**: application/json
 

data/docs/TextApi.md

@@ -0,0 +1,105 @@
+# WebScrapingAI::TextApi
+
+All URIs are relative to *https://api.webscraping.ai*
+
+| Method | HTTP request | Description |
+| ------ | ------------ | ----------- |
+| [**get_text**](TextApi.md#get_text) | **GET** /text | Page text by URL |
+
+
+## get_text
+
+> String get_text(url, opts)
+
+Page text by URL
+
+Returns the visible text content of a webpage specified by the URL. Can be used to feed data to GPT or other LLM models. The response can be in plain text, JSON, or XML format based on the text_format parameter. Proxies and Chromium JavaScript rendering are used for page retrieval and processing. Returns JSON on error.
+
+### Examples
+
+```ruby
+require 'time'
+require 'webscraping_ai'
+# setup authorization
+WebScrapingAI.configure do |config|
+  # Configure API key authorization: api_key
+  config.api_key['api_key'] = 'YOUR API KEY'
+  # Uncomment the following line to set a prefix for the API key, e.g. 'Bearer' (defaults to nil)
+  # config.api_key_prefix['api_key'] = 'Bearer'
+end
+
+api_instance = WebScrapingAI::TextApi.new
+url = 'https://example.com' # String | URL of the target page.
+opts = {
+  text_format: 'plain', # String | Format of the text response (plain by default). \"plain\" will return only the page body text. \"json\" and \"xml\" will return a json/xml with \"title\", \"description\" and \"content\" keys.
+  return_links: false, # Boolean | [Works only with text_format=json] Return links from the page body text (false by default). Useful for building web crawlers.
+  headers: { key: 3.56}, # Hash<String, String> | HTTP headers to pass to the target page. Can be specified either via a nested query parameter (...&headers[One]=value1&headers=[Another]=value2) or as a JSON encoded object (...&headers={\"One\": \"value1\", \"Another\": \"value2\"}).
+  timeout: 10000, # Integer | Maximum web page retrieval time in ms. Increase it in case of timeout errors (10000 by default, maximum is 30000).
+  js: true, # Boolean | Execute on-page JavaScript using a headless browser (true by default).
+  js_timeout: 2000, # Integer | Maximum JavaScript rendering time in ms. Increase it in case if you see a loading indicator instead of data on the target page.
+  proxy: 'datacenter', # String | Type of proxy, use residential proxies if your site restricts traffic from datacenters (datacenter by default). Note that residential proxy requests are more expensive than datacenter, see the pricing page for details.
+  country: 'us', # String | Country of the proxy to use (US by default). Only available on Startup and Custom plans.
+  device: 'desktop', # String | Type of device emulation.
+  error_on_404: false, # Boolean | Return error on 404 HTTP status on the target page (false by default).
+  error_on_redirect: false, # Boolean | Return error on redirect on the target page (false by default).
+  js_script: 'document.querySelector('button').click();' # String | Custom JavaScript code to execute on the target page.
+}
+
+begin
+  # Page text by URL
+  result = api_instance.get_text(url, opts)
+  p result
+rescue WebScrapingAI::ApiError => e
+  puts "Error when calling TextApi->get_text: #{e}"
+end
+```
+
+#### Using the get_text_with_http_info variant
+
+This returns an Array which contains the response data, status code and headers.
+
+> <Array(String, Integer, Hash)> get_text_with_http_info(url, opts)
+
+```ruby
+begin
+  # Page text by URL
+  data, status_code, headers = api_instance.get_text_with_http_info(url, opts)
+  p status_code # => 2xx
+  p headers # => { ... }
+  p data # => String
+rescue WebScrapingAI::ApiError => e
+  puts "Error when calling TextApi->get_text_with_http_info: #{e}"
+end
+```
+
+### Parameters
+
+| Name | Type | Description | Notes |
+| ---- | ---- | ----------- | ----- |
+| **url** | **String** | URL of the target page. |  |
+| **text_format** | **String** | Format of the text response (plain by default). \&quot;plain\&quot; will return only the page body text. \&quot;json\&quot; and \&quot;xml\&quot; will return a json/xml with \&quot;title\&quot;, \&quot;description\&quot; and \&quot;content\&quot; keys. | [optional][default to &#39;plain&#39;] |
+| **return_links** | **Boolean** | [Works only with text_format&#x3D;json] Return links from the page body text (false by default). Useful for building web crawlers. | [optional][default to false] |
+| **headers** | [**Hash&lt;String, String&gt;**](String.md) | HTTP headers to pass to the target page. Can be specified either via a nested query parameter (...&amp;headers[One]&#x3D;value1&amp;headers&#x3D;[Another]&#x3D;value2) or as a JSON encoded object (...&amp;headers&#x3D;{\&quot;One\&quot;: \&quot;value1\&quot;, \&quot;Another\&quot;: \&quot;value2\&quot;}). | [optional] |
+| **timeout** | **Integer** | Maximum web page retrieval time in ms. Increase it in case of timeout errors (10000 by default, maximum is 30000). | [optional][default to 10000] |
+| **js** | **Boolean** | Execute on-page JavaScript using a headless browser (true by default). | [optional][default to true] |
+| **js_timeout** | **Integer** | Maximum JavaScript rendering time in ms. Increase it in case if you see a loading indicator instead of data on the target page. | [optional][default to 2000] |
+| **proxy** | **String** | Type of proxy, use residential proxies if your site restricts traffic from datacenters (datacenter by default). Note that residential proxy requests are more expensive than datacenter, see the pricing page for details. | [optional][default to &#39;datacenter&#39;] |
+| **country** | **String** | Country of the proxy to use (US by default). Only available on Startup and Custom plans. | [optional][default to &#39;us&#39;] |
+| **device** | **String** | Type of device emulation. | [optional][default to &#39;desktop&#39;] |
+| **error_on_404** | **Boolean** | Return error on 404 HTTP status on the target page (false by default). | [optional][default to false] |
+| **error_on_redirect** | **Boolean** | Return error on redirect on the target page (false by default). | [optional][default to false] |
+| **js_script** | **String** | Custom JavaScript code to execute on the target page. | [optional] |
+
+### Return type
+
+**String**
+
+### Authorization
+
+[api_key](../README.md#api_key)
+
+### HTTP request headers
+
+- **Content-Type**: Not defined
+- **Accept**: application/json, text/html, text/xml
+

data/git_push.sh

@@ -1,7 +1,7 @@
 #!/bin/sh
 # ref: https://help.github.com/articles/adding-an-existing-project-to-github-using-the-command-line/
 #
-# Usage example: /bin/sh ./git_push.sh wing328 openapi-pestore-perl "minor update" "gitlab.com"
+# Usage example: /bin/sh ./git_push.sh wing328 openapi-petstore-perl "minor update" "gitlab.com"
 
 git_user_id=$1
 git_repo_id=$2
@@ -38,14 +38,14 @@ git add .
 git commit -m "$release_note"
 
 # Sets the new remote
-git_remote=`git remote`
+git_remote=$(git remote)
 if [ "$git_remote" = "" ]; then # git remote not defined
 
     if [ "$GIT_TOKEN" = "" ]; then
         echo "[INFO] \$GIT_TOKEN (environment variable) is not set. Using the git credential in your environment."
         git remote add origin https://${git_host}/${git_user_id}/${git_repo_id}.git
     else
-        git remote add origin https://${git_user_id}:${GIT_TOKEN}@${git_host}/${git_user_id}/${git_repo_id}.git
+        git remote add origin https://${git_user_id}:"${GIT_TOKEN}"@${git_host}/${git_user_id}/${git_repo_id}.git
     fi
 
 fi
@@ -55,4 +55,3 @@ git pull origin master
 # Pushes (Forces) the changes in the local repository up to the remote repository
 echo "Git pushing to https://${git_host}/${git_user_id}/${git_repo_id}.git"
 git push origin master 2>&1 | grep -v 'To https'
-

data/lib/webscraping_ai/api/account_api.rb

@@ -0,0 +1,79 @@
+=begin
+#WebScraping.AI
+
+#WebScraping.AI scraping API provides GPT-powered tools with Chromium JavaScript rendering, rotating proxies, and built-in HTML parsing.
+
+The version of the OpenAPI document: 3.1.3
+Contact: support@webscraping.ai
+Generated by: https://openapi-generator.tech
+OpenAPI Generator version: 7.2.0
+
+=end
+
+require 'cgi'
+
+module WebScrapingAI
+  class AccountApi
+    attr_accessor :api_client
+
+    def initialize(api_client = ApiClient.default)
+      @api_client = api_client
+    end
+    # Information about your account calls quota
+    # Returns information about your account, including the remaining API credits quota, the next billing cycle start time, and the remaining concurrent requests. The response is in JSON format.
+    # @param [Hash] opts the optional parameters
+    # @return [Account]
+    def account(opts = {})
+      data, _status_code, _headers = account_with_http_info(opts)
+      data
+    end
+
+    # Information about your account calls quota
+    # Returns information about your account, including the remaining API credits quota, the next billing cycle start time, and the remaining concurrent requests. The response is in JSON format.
+    # @param [Hash] opts the optional parameters
+    # @return [Array<(Account, Integer, Hash)>] Account data, response status code and response headers
+    def account_with_http_info(opts = {})
+      if @api_client.config.debugging
+        @api_client.config.logger.debug 'Calling API: AccountApi.account ...'
+      end
+      # resource path
+      local_var_path = '/account'
+
+      # query parameters
+      query_params = opts[:query_params] || {}
+
+      # header parameters
+      header_params = opts[:header_params] || {}
+      # HTTP header 'Accept' (if needed)
+      header_params['Accept'] = @api_client.select_header_accept(['application/json'])
+
+      # form parameters
+      form_params = opts[:form_params] || {}
+
+      # http body (model)
+      post_body = opts[:debug_body]
+
+      # return_type
+      return_type = opts[:debug_return_type] || 'Account'
+
+      # auth_names
+      auth_names = opts[:debug_auth_names] || ['api_key']
+
+      new_options = opts.merge(
+        :operation => :"AccountApi.account",
+        :header_params => header_params,
+        :query_params => query_params,
+        :form_params => form_params,
+        :body => post_body,
+        :auth_names => auth_names,
+        :return_type => return_type
+      )
+
+      data, status_code, headers = @api_client.call_api(:GET, local_var_path, new_options)
+      if @api_client.config.debugging
+        @api_client.config.logger.debug "API called: AccountApi#account\nData: #{data.inspect}\nStatus code: #{status_code}\nHeaders: #{headers}"
+      end
+      return data, status_code, headers
+    end
+  end
+end