proxy_fetcher 0.4.1 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 40029164cd3a21183e8f22191e2cd8c01ba05ae3
-  data.tar.gz: ff557370ec0e68c0817fb009f9e483f936eaecd4
+  metadata.gz: 287db7b55e3f0798e263fe7268f8a709e4d8e8c0
+  data.tar.gz: cde6d4dc22e60aa012c02b1f679fcc72b23c6114
 SHA512:
-  metadata.gz: 9a1461d4c42c3478675a4ad20e0d5e41a674d3486fe1026fdf4d95a1e484c4048f8f01476aaac14ccfa0c2a9e4e78ad2049e6b284828eeeb7c6670b1d0923630
-  data.tar.gz: 98f74fc50f15dc7f21f0f308494e4e874ad184cad512b401173da7ecc648c4fd0672aca45f2f3a4f0ec55aa6c01850d7979b579b8609a013d11c37a59443c10b
+  metadata.gz: a54e2d725338bc5c859d415cae5b0397b4aa2a75a0f00917edf4eaa2f845e893c1d73f8e56b86ddfce85c6016fcf473ed4ad2e29fbf78a2c65495df050a320d7
+  data.tar.gz: 61b2fc3dfd20c75045c8b435d0855e0596071facc8bef9e73d68e46996faaad8a9e1ccbb3b9b818f1449138e7d52ee583dffaa813cc5d3696a24ad6ba835475c

data/.gitignore

@@ -14,6 +14,7 @@ pickle-email-*.html
 .idea
 Gemfile.lock
 *.gem
+certs
 
 # TODO Comment out this rule if you are OK with secrets being uploaded to the repo
 config/initializers/secret_token.rb

data/.rubocop.yml

@@ -1,6 +1,7 @@
 LineLength:
   Max: 120
 AllCops:
+  TargetRubyVersion: 2.4
   Exclude:
     - 'spec/**/*'
     - 'bin/*'
@@ -9,3 +10,5 @@ Rails:
   Enabled: false
 Documentation:
   Enabled: false
+FrozenStringLiteralComment:
+  Enabled: false

data/CHANGELOG.md

@@ -2,6 +2,22 @@
 
 Reverse Chronological Order:
 
+## `0.5.0` (2017-09-06)
+
+* Remove HideMyName provider (not works anymore)
+* Fix ProxyDocker provider
+* Add `ProxyFetcher::Client` to make interacting with proxies easier
+* Add new providers (Gather Proxy & HTTP Tunnel Genius)
+* Simplify `connection_timeout` config option to `timeout`
+* Make User-Agent configurable
+* Move all the gem exceptions under `ProxyFetcher::Error` base class
+* Small improvements
+
+## `0.4.1` (2017-09-04)
+
+* Use all registered providers by default
+* Disable HideMyName provider (now ше uses CloudFlare)
+
 ## `0.4.0` (2017-08-26)
 
 * Support operations with multiple providers

data/Gemfile

@@ -4,4 +4,5 @@ gemspec
 
 group :test do
   gem 'coveralls', require: false
+  gem 'evil-proxy'
 end

data/README.md

@@ -6,7 +6,7 @@
 [![License](http://img.shields.io/badge/license-MIT-brightgreen.svg)](#license)
 
 This gem can help your Ruby application to make HTTP(S) requests from proxy by fetching and validating actual
-proxy lists from multiple providers like [HideMyName](https://hidemy.name/en/).
+proxy lists from multiple providers.
 
 It gives you a `Manager` class that can load proxy lists, validate them and return random or specific proxies. Take a look
 at the documentation below to find all the gem features.
@@ -20,6 +20,7 @@ validating proxy lists from the different providers. [Checkout examples](#standa
 - [Example of usage](#example-of-usage)
   - [In Ruby application](#in-ruby-application)
   - [Standalone](#standalone)
+- [Client](#client)
 - [Configuration](#configuration)
   - [Proxy validation speed](#proxy-validation-speed)
 - [Proxy object](#proxy-object)
@@ -32,7 +33,7 @@ validating proxy lists from the different providers. [Checkout examples](#standa
 If using bundler, first add 'proxy_fetcher' to your Gemfile:
 
 ```ruby
-gem 'proxy_fetcher', '~> 0.4'
+gem 'proxy_fetcher', '~> 0.5'
 ```
 
 or if you want to use the latest version (from `master` branch), then:
@@ -50,7 +51,7 @@ bundle install
 Otherwise simply install the gem:
 
 ```sh
-gem install proxy_fetcher -v '0.4'
+gem install proxy_fetcher -v '0.5'
 ```
 
 ## Example of usage
@@ -123,7 +124,7 @@ If you need to filter proxy list, for example, by country or response time and s
 then you can just pass your filters like a simple Ruby hash to the Manager instance:
 
 ```ruby
-ProxyFetcher.config.providers = :hide_my_name
+ProxyFetcher.config.providers = :proxy_docker
 
 manager = ProxyFetcher::Manager.new(filters: { country: 'PL', maxtime: '500' })
 manager.proxies
@@ -134,7 +135,7 @@ manager.proxies
 If you are using multiple providers, then you can split your filters by proxy provider names:
 
 ```ruby
-ProxyFetcher.config.providers = [:hide_my_name, :xroxy]
+ProxyFetcher.config.providers = [:proxy_docker, :xroxy]
 
 manager = ProxyFetcher::Manager.new(filters: {
   hide_my_name: {
@@ -194,19 +195,64 @@ To get all the possible options run:
 proxy_fetcher --help
 ```
 
+## Client
+
+ProxyFetcher gem provides you a ready-to-use HTTP client that made requesting with proxies easy. It does all the work 
+with the proxy lists for you (load, validate, refresh, find proxy by type, follow redirects, etc). All you need it to
+make HTTP(S) requests:
+
+```ruby
+require 'proxy-fetcher'
+
+ProxyFetcher::Client.get 'https://example.com/resource'
+
+ProxyFetcher::Client.post 'https://example.com/resource', { param: 'value' }
+
+ProxyFetcher::Client.post 'https://example.com/resource', 'Any data'
+
+ProxyFetcher::Client.post 'https://example.com/resource', { param: 'value'}.to_json , headers: { 'Content-Type': 'application/json' }
+
+ProxyFetcher::Client.put 'https://example.com/resource', { param: 'value' }
+
+ProxyFetcher::Client.patch 'https://example.com/resource', { param: 'value' }
+
+ProxyFetcher::Client.delete 'https://example.com/resource'
+```
+
+By default, `ProxyFetcher::Client` makes 1000 attempts to send a HTTP request in case if proxy is out of order or the
+remote server returns an error. You can increase or decrease this number for your case or set it to `nil` if you want to
+make infinite number of requests (or before your Ruby process will die :skull:):
+
+```ruby
+require 'proxy-fetcher'
+
+ProxyFetcher::Client.get 'https://example.com/resource', options: { max_retries: 10_000 }
+```
+
+Btw, if you need support of JavaScript or some other features, you need to implement your own client using, for example,
+`selenium-webdriver`.
+
 ## Configuration
 
 To change open/read timeout for `cleanup!` and `connectable?` methods you need to change `ProxyFetcher.config`:
 
 ```ruby
 ProxyFetcher.configure do |config|
-  config.connection_timeout = 1 # default is 3
+  config.timeout = 1 # default is 3
 end
 
 manager = ProxyFetcher::Manager.new
 manager.cleanup!
 ```
 
+Also you can set your custom User-Agent:
+
+```ruby
+ProxyFetcher.configure do |config|
+  config.user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'
+end
+```
+
 ProxyFetcher uses simple Ruby solution for dealing with HTTP(S) requests - `net/http` library from the stdlib. If you wanna add, for example, your custom provider that
 was developed as a Single Page Application (SPA) with some JavaScript, then you will need something like [selenium-webdriver](https://github.com/SeleniumHQ/selenium/tree/master/rb)
 to properly load the content of the website. For those and other cases you can write your own class for fetching HTML content by the URL and setup it
@@ -269,7 +315,7 @@ ProxyFetcher.config.pool_size = 50
 You can experiment with the threads pool size to find an optimal number of maximum threads count for you PC and OS.
 This will definitely give you some performance improvements.
 
-Moreover, the common proxy validation speed depends on `ProxyFetcher.config.connection_timeout` option that is equal
+Moreover, the common proxy validation speed depends on `ProxyFetcher.config.timeout` option that is equal
 to `3` by default. It means that gem will wait 3 seconds for the server answer to check if particular proxy is connectable.
 You can decrease this option to `1`, for example, and it will heavily increase proxy validation speed (**but remember**
 that some proxies could be connectable, but slow, so with this option you will clear proxy list from the proxies that
@@ -300,10 +346,11 @@ Also you can call next instance methods for every Proxy object:
 
 Currently ProxyFetcher can deal with next proxy providers (services):
 
-* Hide My Name (**currently does not work**)
 * Free Proxy List
 * Free SSL Proxies
 * Proxy Docker
+* Gather Proxy
+* HTTP Tunnel Genius
 * Proxy List
 * XRoxy
 

data/lib/proxy_fetcher.rb

@@ -3,6 +3,7 @@ require 'net/https'
 require 'nokogiri'
 require 'thread'
 
+require File.dirname(__FILE__) + '/proxy_fetcher/exceptions'
 require File.dirname(__FILE__) + '/proxy_fetcher/configuration'
 require File.dirname(__FILE__) + '/proxy_fetcher/configuration/providers_registry'
 require File.dirname(__FILE__) + '/proxy_fetcher/proxy'
@@ -11,13 +12,17 @@ require File.dirname(__FILE__) + '/proxy_fetcher/manager'
 require File.dirname(__FILE__) + '/proxy_fetcher/utils/http_client'
 require File.dirname(__FILE__) + '/proxy_fetcher/utils/html'
 require File.dirname(__FILE__) + '/proxy_fetcher/utils/proxy_validator'
+require File.dirname(__FILE__) + '/proxy_fetcher/client/client'
+require File.dirname(__FILE__) + '/proxy_fetcher/client/request'
+require File.dirname(__FILE__) + '/proxy_fetcher/client/proxies_registry'
 
 module ProxyFetcher
   module Providers
     require File.dirname(__FILE__) + '/proxy_fetcher/providers/base'
     require File.dirname(__FILE__) + '/proxy_fetcher/providers/free_proxy_list'
     require File.dirname(__FILE__) + '/proxy_fetcher/providers/free_proxy_list_ssl'
-    require File.dirname(__FILE__) + '/proxy_fetcher/providers/hide_my_name'
+    require File.dirname(__FILE__) + '/proxy_fetcher/providers/gather_proxy'
+    require File.dirname(__FILE__) + '/proxy_fetcher/providers/http_tunnel'
     require File.dirname(__FILE__) + '/proxy_fetcher/providers/proxy_docker'
     require File.dirname(__FILE__) + '/proxy_fetcher/providers/proxy_list'
     require File.dirname(__FILE__) + '/proxy_fetcher/providers/xroxy'

data/lib/proxy_fetcher/client/client.rb

@@ -0,0 +1,71 @@
+module ProxyFetcher
+  module Client
+    class << self
+      def get(url, headers: {}, options: {})
+        request_without_payload(:get, url, headers, options)
+      end
+
+      def head(url, headers: {}, options: {})
+        request_without_payload(:head, url, headers, options)
+      end
+
+      def post(url, payload, headers: {}, options: {})
+        request_with_payload(:post, url, payload, headers, options)
+      end
+
+      def delete(url, headers: {}, options: {})
+        request_without_payload(:delete, url, headers, options)
+      end
+
+      def put(url, payload, headers: {}, options: {})
+        request_with_payload(:put, url, payload, headers, options)
+      end
+
+      def patch(url, payload, headers: {}, options: {})
+        request_with_payload(:patch, url, payload, headers, options)
+      end
+
+      private
+
+      def request_with_payload(method, url, payload, headers, options)
+        safe_request_to(url, options.fetch(:max_retries, 1000)) do |proxy|
+          opts = options.merge(url: url, payload: payload, proxy: proxy, headers: default_headers.merge(headers))
+
+          Request.execute(method: method, **opts)
+        end
+      end
+
+      def request_without_payload(method, url, headers, options)
+        safe_request_to(url, options.fetch(:max_retries, 1000)) do |proxy|
+          opts = options.merge(url: url, proxy: proxy, headers: default_headers.merge(headers))
+
+          Request.execute(method: method, **opts)
+        end
+      end
+
+      def default_headers
+        {
+          'User-Agent' => ProxyFetcher.config.user_agent
+        }
+      end
+
+      def safe_request_to(url, max_retries = 1000)
+        tries = 0
+
+        begin
+          proxy = ProxiesRegistry.find_proxy_for(url)
+          yield(proxy)
+        rescue ProxyFetcher::Error
+          raise
+        rescue StandardError
+          raise ProxyFetcher::Exceptions::MaximumRetriesReached if max_retries && tries >= max_retries
+
+          ProxiesRegistry.invalidate_proxy!(proxy)
+          tries += 1
+
+          retry
+        end
+      end
+    end
+  end
+end

data/lib/proxy_fetcher/client/proxies_registry.rb

@@ -0,0 +1,32 @@
+module ProxyFetcher
+  module Client
+    class ProxiesRegistry
+      class << self
+        def invalidate_proxy!(proxy)
+          manager.proxies.delete(proxy)
+          manager.refresh_list! if manager.proxies.empty?
+        end
+
+        def find_proxy_for(url)
+          proxy = if URI.parse(url).is_a?(URI::HTTPS)
+                    manager.proxies.detect(&:ssl?)
+                  else
+                    manager.get
+                  end
+
+          return proxy unless proxy.nil?
+
+          manager.refresh_list!
+          find_proxy_for(url)
+        end
+
+        def manager
+          manager = Thread.current[:proxy_fetcher_manager]
+          return manager unless manager.nil?
+
+          Thread.current[:proxy_fetcher_manager] = ProxyFetcher::Manager.new
+        end
+      end
+    end
+  end
+end

data/lib/proxy_fetcher/client/request.rb

@@ -0,0 +1,88 @@
+module ProxyFetcher
+  module Client
+    class Request
+      URL_ENCODED = {
+        'Content-Type' => 'application/x-www-form-urlencoded'
+      }.freeze
+
+      DEFAULT_SSL_OPTIONS = {
+        verify_mode: OpenSSL::SSL::VERIFY_NONE
+      }.freeze
+
+      attr_reader :http, :method, :uri, :headers, :timeout,
+                  :payload, :proxy, :max_redirects, :ssl_options
+
+      def self.execute(args)
+        new(args).execute
+      end
+
+      def initialize(args)
+        raise ArgumentError, 'args must be a Hash!' unless args.is_a?(Hash)
+
+        @uri = URI.parse(args.fetch(:url))
+        @method = args.fetch(:method).to_s.capitalize
+        @headers = (args[:headers] || {}).dup
+        @payload = preprocess_payload(args[:payload])
+        @timeout = args.fetch(:timeout, ProxyFetcher.config.timeout)
+        @ssl_options = args.fetch(:ssl_options, DEFAULT_SSL_OPTIONS)
+
+        @proxy = args.fetch(:proxy)
+        @max_redirects = args.fetch(:max_redirects, 10)
+
+        build_http_client
+      end
+
+      def execute
+        request = request_class_for(method).new(uri, headers)
+
+        http.start do |connection|
+          process_response!(connection.request(request, payload))
+        end
+      end
+
+      private
+
+      def preprocess_payload(payload)
+        return if payload.nil?
+
+        if payload.is_a?(Hash)
+          headers.merge(URL_ENCODED)
+          URI.encode_www_form(payload)
+        else
+          payload
+        end
+      end
+
+      def build_http_client
+        @http = Net::HTTP.new(uri.host, uri.port, proxy.addr, proxy.port)
+
+        @http.use_ssl = uri.is_a?(URI::HTTPS)
+        @http.verify_mode = ssl_options.fetch(:verify_mode)
+        @http.open_timeout = timeout
+        @http.read_timeout = timeout
+      end
+
+      def process_response!(http_response)
+        case http_response
+        when Net::HTTPSuccess then http_response.read_body
+        when Net::HTTPRedirection then follow_redirection(http_response)
+        else
+          http_response.error!
+        end
+      end
+
+      def follow_redirection(http_response)
+        raise ProxyFetcher::Exceptions::MaximumRedirectsReached if max_redirects <= 0
+
+        url = http_response.fetch('location')
+        url = uri.merge(url).to_s unless url.downcase.start_with?('http')
+
+        Request.execute(method: :get, url: url, proxy: proxy, headers: headers, timeout: timeout, max_redirects: max_redirects - 1)
+      end
+
+      def request_class_for(method)
+        Net::HTTP.const_get(method, false)
+      end
+    end
+  end
+end

data/lib/proxy_fetcher/configuration.rb

@@ -1,10 +1,11 @@
 module ProxyFetcher
   class Configuration
-    WrongCustomClass = Class.new(StandardError)
-
-    attr_accessor :providers, :connection_timeout, :pool_size
+    attr_accessor :providers, :timeout, :pool_size, :user_agent
     attr_accessor :http_client, :proxy_validator
 
+    # rubocop:disable Metrics/LineLength
+    DEFAULT_USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112 Safari/537.36'.freeze
+
     class << self
       def providers_registry
         @registry ||= ProvidersRegistry.new
@@ -25,8 +26,9 @@ module ProxyFetcher
 
     # Sets default configuration options
     def reset!
+      @user_agent = DEFAULT_USER_AGENT
       @pool_size = 10
-      @connection_timeout = 3
+      @timeout = 3
       @http_client = HTTPClient
       @proxy_validator = ProxyValidator
 
@@ -53,7 +55,7 @@ module ProxyFetcher
     # Checks if custom class has some required class methods
     def setup_custom_class(klass, required_methods: [])
       unless klass.respond_to?(*required_methods)
-        raise WrongCustomClass, "#{klass} must respond to [#{Array(required_methods).join(', ')}] class methods!"
+        raise ProxyFetcher::Exceptions::WrongCustomClass.new(klass, required_methods)
       end
 
       klass

data/lib/proxy_fetcher/configuration/providers_registry.rb

@@ -1,17 +1,14 @@
 module ProxyFetcher
   class ProvidersRegistry
-    UnknownProvider = Class.new(StandardError)
-    RegisteredProvider = Class.new(StandardError)
-
     def providers
       @providers ||= {}
     end
 
     # Add custom provider to common registry.
-    # Requires proxy provider name ('hide_my_name' for example) and a class
+    # Requires proxy provider name ('proxy_docker' for example) and a class
     # that implements the parsing logic.
     def register(name, klass)
-      raise RegisteredProvider, "`#{name}` provider already registered!" if providers.key?(name.to_sym)
+      raise ProxyFetcher::Exceptions::RegisteredProvider, name if providers.key?(name.to_sym)
 
       providers[name.to_sym] = klass
     end
@@ -23,7 +20,7 @@ module ProxyFetcher
 
       providers.fetch(provider_name)
     rescue KeyError
-      raise UnknownProvider, "unregistered proxy provider `#{provider_name}`"
+      raise ProxyFetcher::Exceptions::UnknownProvider, provider_name
     end
   end
 end

data/lib/proxy_fetcher/exceptions.rb

@@ -0,0 +1,36 @@
+module ProxyFetcher
+  Error = Class.new(StandardError)
+
+  module Exceptions
+    class WrongCustomClass < Error
+      def initialize(klass, methods)
+        required_methods = Array(methods).join(', ')
+        super("#{klass} must respond to [#{required_methods}] class methods!")
+      end
+    end
+
+    class UnknownProvider < Error
+      def initialize(provider_name)
+        super("unregistered proxy provider `#{provider_name}`")
+      end
+    end
+
+    class RegisteredProvider < Error
+      def initialize(name)
+        super("`#{name}` provider already registered!")
+      end
+    end
+
+    class MaximumRedirectsReached < Error
+      def initialize(*)
+        super('maximum redirects reached')
+      end
+    end
+
+    class MaximumRetriesReached < Error
+      def initialize(*)
+        super('reached the maximum number of retries')
+      end
+    end
+  end
+end

data/lib/proxy_fetcher/providers/gather_proxy.rb

@@ -0,0 +1,36 @@
+require 'json'
+
+module ProxyFetcher
+  module Providers
+    class GatherProxy < Base
+      PROVIDER_URL = 'http://www.gatherproxy.com/'.freeze
+
+      def load_proxy_list(*)
+        doc = load_document(PROVIDER_URL)
+        doc.xpath('//div[@class="proxy-list"]/table/script')
+      end
+
+      def to_proxy(html_element)
+        json = parse_json(html_element)
+
+        ProxyFetcher::Proxy.new.tap do |proxy|
+          proxy.addr = json['PROXY_IP']
+          proxy.port = json['PROXY_PORT'].to_i(16)
+          proxy.anonymity = json['PROXY_TYPE']
+          proxy.country = json['PROXY_COUNTRY']
+          proxy.response_time = json['PROXY_TIME'].to_i
+          proxy.type = ProxyFetcher::Proxy::HTTP
+        end
+      end
+
+      private
+
+      def parse_json(element)
+        javascript = clear(element.content)[/{.+}/im]
+        JSON.parse(javascript)
+      end
+    end
+
+    ProxyFetcher::Configuration.register_provider(:gather_proxy, GatherProxy)
+  end
+end

data/lib/proxy_fetcher/providers/http_tunnel.rb

@@ -0,0 +1,48 @@
+module ProxyFetcher
+  module Providers
+    class HTTPTunnel < Base
+      PROVIDER_URL = 'http://www.httptunnel.ge/ProxyListForFree.aspx'.freeze
+
+      def load_proxy_list(*)
+        doc = load_document(PROVIDER_URL)
+        doc.xpath('//table[contains(@id, "GridView")]/tr[(count(td)>2)]')
+      end
+
+      def to_proxy(html_element)
+        ProxyFetcher::Proxy.new.tap do |proxy|
+          uri = parse_proxy_uri(html_element)
+          proxy.addr = uri.host
+          proxy.port = uri.port
+
+          proxy.country = parse_country(html_element)
+          proxy.anonymity = parse_anonymity(html_element)
+          proxy.type = ProxyFetcher::Proxy::HTTP
+        end
+      end
+
+      private
+
+      def parse_proxy_uri(element)
+        full_addr = parse_element(element, 'td[1]')
+        URI.parse("http://#{full_addr}")
+      end
+
+      def parse_country(element)
+        element.at('img').attr('title')
+      end
+
+      def parse_anonymity(element)
+        transparency = parse_element(element, 'td[5]').to_sym
+
+        {
+          A: 'Anonimous',
+          E: 'Elite',
+          T: 'Transparent',
+          U: 'Unknown'
+        }.fetch(transparency, 'Unknown')
+      end
+    end
+
+    ProxyFetcher::Configuration.register_provider(:http_tunnel, HTTPTunnel)
+  end
+end

data/lib/proxy_fetcher/providers/proxy_docker.rb

@@ -1,7 +1,7 @@
 module ProxyFetcher
   module Providers
     class ProxyDocker < Base
-      PROVIDER_URL = 'https://www.proxydocker.com/en'.freeze
+      PROVIDER_URL = 'https://www.proxydocker.com/'.freeze
 
       # [NOTE] Doesn't support direct filters
       def load_proxy_list(*)

data/lib/proxy_fetcher/proxy.rb

@@ -15,7 +15,9 @@ module ProxyFetcher
       end
     end
 
-    alias ssl? https?
+    def ssl?
+      https? || socks4? || socks5?
+    end
 
     def initialize(attributes = {})
       attributes.each do |attr, value|

data/lib/proxy_fetcher/utils/http_client.rb

@@ -14,12 +14,13 @@ module ProxyFetcher
     def fetch
       request = Net::HTTP::Get.new(@uri.to_s)
       request['Connection'] = 'keep-alive'
+      request['User-Agent'] = ProxyFetcher.config.user_agent
       response = @http.request(request)
       response.body
     end
 
     def https?
-      @uri.scheme.casecmp('https').zero?
+      @uri.is_a?(URI::HTTPS)
     end
 
     class << self

data/lib/proxy_fetcher/utils/proxy_validator.rb

@@ -6,15 +6,15 @@ module ProxyFetcher
       uri = URI.parse(URL_TO_CHECK)
       @http = Net::HTTP.new(uri.host, uri.port, proxy_addr, proxy_port.to_i)
 
-      return unless uri.scheme.casecmp('https').zero?
+      return unless uri.is_a?(URI::HTTPS)
 
       @http.use_ssl = true
       @http.verify_mode = OpenSSL::SSL::VERIFY_NONE
     end
 
     def connectable?
-      @http.open_timeout = ProxyFetcher.config.connection_timeout
-      @http.read_timeout = ProxyFetcher.config.connection_timeout
+      @http.open_timeout = ProxyFetcher.config.timeout
+      @http.read_timeout = ProxyFetcher.config.timeout
 
       @http.start { |connection| return true if connection.request_head('/') }
 

data/lib/proxy_fetcher/version.rb

@@ -7,9 +7,9 @@ module ProxyFetcher
     # Major version number
     MAJOR = 0
     # Minor version number
-    MINOR = 4
+    MINOR = 5
     # Smallest version number
-    TINY  = 1
+    TINY  = 0
 
     # Full version number
     STRING = [MAJOR, MINOR, TINY].compact.join('.')

data/proxy_fetcher.gemspec

@@ -5,7 +5,7 @@ require 'proxy_fetcher/version'
 Gem::Specification.new do |gem|
   gem.name = 'proxy_fetcher'
   gem.version = ProxyFetcher.gem_version
-  gem.date = '2017-09-04'
+  gem.date = '2017-09-06'
   gem.summary = 'Ruby gem for dealing with proxy lists from different providers'
   gem.description = 'This gem can help your Ruby application to make HTTP(S) requests ' \
                     'from proxy server by fetching and validating proxy lists from the different providers.'

data/spec/proxy_fetcher/client_spec.rb

@@ -0,0 +1,125 @@
+require 'spec_helper'
+require 'json'
+
+require 'evil-proxy'
+require 'evil-proxy/async'
+
+describe ProxyFetcher::Client do
+  before :all do
+    ProxyFetcher.configure do |config|
+      config.provider = :xroxy
+      config.timeout = 5
+    end
+
+    @server = EvilProxy::MITMProxyServer.new Port: 3128, Quiet: true
+    @server.start
+  end
+
+  after :all do
+    @server.shutdown
+  end
+
+  # Use local proxy server in order to avoid side effects, non-working proxies, etc
+  before :each do
+    proxy = ProxyFetcher::Proxy.new(addr: '127.0.0.1', port: 3128, type: 'HTTP, HTTPS')
+    ProxyFetcher::Client::ProxiesRegistry.manager.instance_variable_set(:'@proxies', [proxy])
+    allow_any_instance_of(ProxyFetcher::Providers::Base).to receive(:fetch_proxies!).and_return([proxy])
+  end
+
+  context 'GET request with the valid proxy' do
+    it 'successfully returns page content for HTTP' do
+      content = ProxyFetcher::Client.get('http://httpbin.org')
+
+      expect(content).not_to be_nil
+      expect(content).not_to be_empty
+    end
+
+    it 'successfully returns page content for HTTPS' do
+      content = ProxyFetcher::Client.get('https://httpbin.org')
+
+      expect(content).not_to be_nil
+      expect(content).not_to be_empty
+    end
+  end
+
+  context 'POST request with the valid proxy' do
+    it 'successfully returns page content for HTTP' do
+      headers = {
+        'X-Proxy-Fetcher-Version' => ProxyFetcher::VERSION::STRING
+      }
+      content = ProxyFetcher::Client.post('http://httpbin.org/post', { param: 'value'} , headers: headers)
+
+      expect(content).not_to be_nil
+      expect(content).not_to be_empty
+
+      json = JSON.parse(content)
+
+      expect(json['headers']['X-Proxy-Fetcher-Version']).to eq(ProxyFetcher::VERSION::STRING)
+      expect(json['headers']['User-Agent']).to eq(ProxyFetcher.config.user_agent)
+    end
+  end
+
+  context 'PUT request with the valid proxy' do
+    it 'successfully returns page content for HTTP' do
+      content = ProxyFetcher::Client.put('http://httpbin.org/put', 'param=PutValue')
+
+      expect(content).not_to be_nil
+      expect(content).not_to be_empty
+
+      json = JSON.parse(content)
+
+      expect(json['form']['param']).to eq('PutValue')
+    end
+  end
+
+  context 'PATCH request with the valid proxy' do
+    it 'successfully returns page content for HTTP' do
+      content = ProxyFetcher::Client.patch('http://httpbin.org/patch', param: 'value')
+
+      expect(content).not_to be_nil
+      expect(content).not_to be_empty
+
+      json = JSON.parse(content)
+
+      expect(json['form']['param']).to eq('value')
+    end
+  end
+
+  context 'DELETE request with the valid proxy' do
+    it 'successfully returns page content for HTTP' do
+      content = ProxyFetcher::Client.delete('http://httpbin.org/delete')
+
+      expect(content).not_to be_nil
+      expect(content).not_to be_empty
+    end
+  end
+
+  context 'HEAD request with the valid proxy' do
+    it 'successfully works' do
+      content = ProxyFetcher::Client.head('http://httpbin.org')
+
+      expect(content).to be_nil
+    end
+  end
+
+  context 'retries' do
+    it 'raises an error when reaches max retries limit' do
+      allow(ProxyFetcher::Client::Request).to receive(:execute).and_raise(StandardError)
+
+      expect { ProxyFetcher::Client.get('http://httpbin.org') }.to raise_error(ProxyFetcher::Exceptions::MaximumRetriesReached)
+    end
+  end
+
+  context 'redirects' do
+    it 'follows redirect when present' do
+      content = ProxyFetcher::Client.get('http://httpbin.org/absolute-redirect/2')
+
+      expect(content).not_to be_nil
+      expect(content).not_to be_empty
+    end
+
+    it 'raises an error when reaches max redirects limit' do
+      expect { ProxyFetcher::Client.get('http://httpbin.org/absolute-redirect/11') }.to raise_error(ProxyFetcher::Exceptions::MaximumRedirectsReached)
+    end
+  end
+end

data/spec/proxy_fetcher/configuration_spec.rb

@@ -19,7 +19,7 @@ describe ProxyFetcher::Configuration do
       MyWrongHTTPClient = Class.new
 
       expect { ProxyFetcher.config.http_client = MyWrongHTTPClient }
-        .to raise_error(ProxyFetcher::Configuration::WrongCustomClass)
+        .to raise_error(ProxyFetcher::Exceptions::WrongCustomClass)
     end
   end
 
@@ -38,21 +38,21 @@ describe ProxyFetcher::Configuration do
       MyWrongProxyValidator = Class.new
 
       expect { ProxyFetcher.config.proxy_validator = MyWrongProxyValidator }
-        .to raise_error(ProxyFetcher::Configuration::WrongCustomClass)
+        .to raise_error(ProxyFetcher::Exceptions::WrongCustomClass)
     end
   end
 
   context 'custom provider' do
     it 'failed on registration if provider class already registered' do
       expect { ProxyFetcher::Configuration.register_provider(:xroxy, Class.new) }
-        .to raise_error(ProxyFetcher::ProvidersRegistry::RegisteredProvider)
+        .to raise_error(ProxyFetcher::Exceptions::RegisteredProvider)
     end
 
     it "failed on proxy list fetching if provider doesn't registered" do
       ProxyFetcher.config.provider = :not_existing_provider
 
       expect { ProxyFetcher::Manager.new }
-        .to raise_error(ProxyFetcher::ProvidersRegistry::UnknownProvider)
+        .to raise_error(ProxyFetcher::Exceptions::UnknownProvider)
     end
   end
 end

data/spec/proxy_fetcher/providers/gather_proxy_spec.rb

@@ -0,0 +1,9 @@
+require 'spec_helper'
+
+describe ProxyFetcher::Providers::GatherProxy do
+  before :all do
+    ProxyFetcher.config.provider = :gather_proxy
+  end
+
+  it_behaves_like 'a manager'
+end

data/spec/proxy_fetcher/providers/http_tunnel_spec.rb

@@ -0,0 +1,9 @@
+require 'spec_helper'
+
+describe ProxyFetcher::Providers::HTTPTunnel do
+  before :all do
+    ProxyFetcher.config.provider = :http_tunnel
+  end
+
+  it_behaves_like 'a manager'
+end

data/spec/spec_helper.rb

@@ -1,8 +1,10 @@
+require 'simplecov'
+SimpleCov.add_filter 'spec'
+
 if ENV['CI'] || ENV['TRAVIS'] || ENV['COVERALLS'] || ENV['JENKINS_URL']
   require 'coveralls'
   Coveralls.wear!
 else
-  require 'simplecov'
   SimpleCov.start
 end
 

data/spec/support/evil_proxy_patch.rb

@@ -0,0 +1,26 @@
+require 'evil-proxy'
+
+EvilProxy::HTTPProxyServer.class_eval do
+  def do_PUT(req, res)
+    perform_proxy_request(req, res) do |http, path, header|
+      http.put(path, req.body || '', header)
+    end
+  end
+
+  def do_DELETE(req, res)
+    perform_proxy_request(req, res) do |http, path, header|
+      http.delete(path, header)
+    end
+  end
+
+  def do_PATCH(req, res)
+    perform_proxy_request(req, res) do |http, path, header|
+      http.patch(path, req.body || '', header)
+    end
+  end
+
+# This method is not needed for PUT but I added for completeness
+  def do_OPTIONS(_req, res)
+    res['allow'] = 'GET,HEAD,POST,OPTIONS,CONNECT,PUT,PATCH,DELETE'
+  end
+end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: proxy_fetcher
 version: !ruby/object:Gem::Version
-  version: 0.4.1
+  version: 0.5.0
 platform: ruby
 authors:
 - Nikita Bulai
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-09-04 00:00:00.000000000 Z
+date: 2017-09-06 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: nokogiri
@@ -63,13 +63,18 @@ files:
 - Rakefile
 - bin/proxy_fetcher
 - lib/proxy_fetcher.rb
+- lib/proxy_fetcher/client/client.rb
+- lib/proxy_fetcher/client/proxies_registry.rb
+- lib/proxy_fetcher/client/request.rb
 - lib/proxy_fetcher/configuration.rb
 - lib/proxy_fetcher/configuration/providers_registry.rb
+- lib/proxy_fetcher/exceptions.rb
 - lib/proxy_fetcher/manager.rb
 - lib/proxy_fetcher/providers/base.rb
 - lib/proxy_fetcher/providers/free_proxy_list.rb
 - lib/proxy_fetcher/providers/free_proxy_list_ssl.rb
-- lib/proxy_fetcher/providers/hide_my_name.rb
+- lib/proxy_fetcher/providers/gather_proxy.rb
+- lib/proxy_fetcher/providers/http_tunnel.rb
 - lib/proxy_fetcher/providers/proxy_docker.rb
 - lib/proxy_fetcher/providers/proxy_list.rb
 - lib/proxy_fetcher/providers/xroxy.rb
@@ -79,17 +84,20 @@ files:
 - lib/proxy_fetcher/utils/proxy_validator.rb
 - lib/proxy_fetcher/version.rb
 - proxy_fetcher.gemspec
+- spec/proxy_fetcher/client_spec.rb
 - spec/proxy_fetcher/configuration_spec.rb
 - spec/proxy_fetcher/providers/base_spec.rb
 - spec/proxy_fetcher/providers/free_proxy_list_spec.rb
 - spec/proxy_fetcher/providers/free_proxy_list_ssl_spec.rb
-- spec/proxy_fetcher/providers/hide_my_name_spec.rb
+- spec/proxy_fetcher/providers/gather_proxy_spec.rb
+- spec/proxy_fetcher/providers/http_tunnel_spec.rb
 - spec/proxy_fetcher/providers/multiple_providers_spec.rb
 - spec/proxy_fetcher/providers/proxy_docker_spec.rb
 - spec/proxy_fetcher/providers/proxy_list_spec.rb
 - spec/proxy_fetcher/providers/xroxy_spec.rb
 - spec/proxy_fetcher/proxy_spec.rb
 - spec/spec_helper.rb
+- spec/support/evil_proxy_patch.rb
 - spec/support/manager_examples.rb
 homepage: http://github.com/nbulaj/proxy_fetcher
 licenses:

data/lib/proxy_fetcher/providers/hide_my_name.rb

@@ -1,35 +0,0 @@
-module ProxyFetcher
-  module Providers
-    class HideMyName < Base
-      PROVIDER_URL = 'https://hidemy.name/en/proxy-list/'.freeze
-
-      def load_proxy_list(filters = { type: 'hs' })
-        doc = load_document(PROVIDER_URL, filters)
-        doc.xpath('//table[@class="proxy__t"]/tbody/tr')
-      end
-
-      def to_proxy(html_element)
-        ProxyFetcher::Proxy.new.tap do |proxy|
-          proxy.addr = parse_element(html_element, 'td[1]')
-          proxy.port = convert_to_int(parse_element(html_element, 'td[2]'))
-          proxy.anonymity = parse_element(html_element, 'td[6]')
-          proxy.country = parse_country(html_element)
-          proxy.type = parse_element(html_element, 'td[5]')
-          proxy.response_time = parse_response_time(html_element)
-        end
-      end
-
-      private
-
-      def parse_country(element)
-        clear(element.at_xpath('*//span[1]/following-sibling::text()[1]').content)
-      end
-
-      def parse_response_time(element)
-        convert_to_int(element.at_xpath('td[4]').content.strip[/\d+/])
-      end
-    end
-
-    ProxyFetcher::Configuration.register_provider(:hide_my_name, HideMyName)
-  end
-end

data/spec/proxy_fetcher/providers/hide_my_name_spec.rb

@@ -1,10 +0,0 @@
-require 'spec_helper'
-
-describe ProxyFetcher::Providers::HideMyName do
-  before :all do
-    ProxyFetcher.config.provider = :hide_my_name
-  end
-
-  # TODO: fix provider
-  # it_behaves_like 'a manager'
-end