proxy_fetcher 0.2.5 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: a0c631e566e3330cda9a53d1ecbaa38cc0c4e055
-  data.tar.gz: e60250b46b1db6cc1f1d77efa8351b8b422c1f7e
+  metadata.gz: 2082cc216a388f9014cbdec5daa8a54ac1d93016
+  data.tar.gz: 6d0d817f9b1fecdbc3512440803a4444bcaf3c67
 SHA512:
-  metadata.gz: a0a8e44ed225617a7110933054111824bfacbc7f32698bb465a51d85b1870a23ac0cf792f9d1af0128f4bba1eb489ee68288317807b8b631cef2900a4e8ae1e6
-  data.tar.gz: 525adfd9031c16d5f709405c1eb70e0dac13e285709e59afb1d6c20755fe3c580dd011a5c773cc35defce5c3dbf565261b4aa23e3a74ffe987afa60733e985d5
+  metadata.gz: f3a33d56dd95b4c3a7755a8f76f780b702af03de5a546b78cf3bfa56fad96b877169d02ca4c4f83dcfdcee8ee7a1efd67983ead21580b0ea4bb94c78db92da3a
+  data.tar.gz: b0019c53bed440256de36853a9b9973f58bd2fd019e90ca1b8178c727ad64572a3a0b3fc4ded49c64d6f4134935b72f124ba4c8780f9ed01b3412bb91c953499

data/README.md

@@ -2,6 +2,7 @@
 [![Gem Version](https://badge.fury.io/rb/proxy_fetcher.svg)](http://badge.fury.io/rb/proxy_fetcher)
 [![Build Status](https://travis-ci.org/nbulaj/proxy_fetcher.svg?branch=master)](https://travis-ci.org/nbulaj/proxy_fetcher)
 [![Coverage Status](https://coveralls.io/repos/github/nbulaj/proxy_fetcher/badge.svg)](https://coveralls.io/github/nbulaj/proxy_fetcher)
+[![Code Climate](https://codeclimate.com/github/nbulaj/proxy_fetcher/badges/gpa.svg)](https://codeclimate.com/github/nbulaj/proxy_fetcher)
 [![License](http://img.shields.io/badge/license-MIT-brightgreen.svg)](#license)
 
 This gem can help your Ruby application to make HTTP(S) requests from proxy by fetching and validating actual
@@ -10,12 +11,15 @@ proxy lists from the different providers like [HideMyName](https://hidemy.name/e
 It gives you a `Manager` class that can load proxy list, validate it and return random or specific proxy entry. Take a look
 at the documentation below to find all the gem features.
 
+Also this gem can be used as standalone solution for downloading and validating proxy lists from the different providers.
+Checkout examples of usage below.
+
 ## Installation
 
 If using bundler, first add 'proxy_fetcher' to your Gemfile:
 
 ```ruby
-gem 'proxy_fetcher', '~> 0.2'
+gem 'proxy_fetcher', '~> 0.3'
 ```
 
 or if you want to use the latest version (from `master` branch), then:
@@ -33,12 +37,14 @@ bundle install
 Otherwise simply install the gem:
 
 ```sh
-gem install proxy_fetcher -v '0.2'
+gem install proxy_fetcher -v '0.3'
 ```
 
 ## Example of usage
 
-Get current proxy list:
+### In Ruby application
+
+Get current proxy list without validation:
 
 ```ruby
 manager = ProxyFetcher::Manager.new # will immediately load proxy list from the server
@@ -48,7 +54,7 @@ manager.proxies
  #     @response_time=5217, @speed=48, @type="HTTP", @anonymity="High">, ... ]
 ```
 
-You can initialize proxy manager without loading proxy list from the remote server by passing `refresh: false` on initialization:
+You can initialize proxy manager without immediate load of proxy list from the remote server by passing `refresh: false` on initialization:
 
 ```ruby
 manager = ProxyFetcher::Manager.new(refresh: false) # just initialize class instance
@@ -57,7 +63,13 @@ manager.proxies
  #=> []
 ```
 
-Get raw proxy URLs:
+If you wanna clean current proxy list from some dead servers that does not respond to the requests, than you can just call `cleanup!` method:
+
+```ruby
+manager.cleanup! # or manager.validate!
+```
+
+Get raw proxy URLs as Strings:
 
 ```ruby
 manager = ProxyFetcher::Manager.new
@@ -76,6 +88,58 @@ manager.refresh_list! # or manager.fetch!
  #     @response_time=5217, @speed=48, @type="HTTP", @anonymity="High">, ... ]
 ```
 
+If you need to filter proxy list, for example, by country or response time and selected provider supports filtering by GET params, then you
+can pass your filters to the Manager instance like that:
+
+```ruby
+ProxyFetcher.config.provider = :hide_my_name
+
+manager = ProxyFetcher::Manager.new(filters: { country: 'AO', maxtime: '500' })
+manager.proxies
+
+ # => [...]
+```
+
+*NOTE*: not all the providers support filtering. Take a look at the provider class to see if it supports custom filters.
+
+You can use two methods to get the first proxy from the list:
+
+* `get` or aliased `pop` (will return first proxy and move it to the end of the list)
+* `get!` or aliased `pop!` (will return first **connectable** proxy and move it to the end of the list; all the proxies till the working one will be removed)
+
+Or you can get just random proxy by calling `manager.random_proxy` or it's alias `manager.random`.
+
+### Standalone
+
+All you need to use this gem is Ruby >= 2.0 (2.3 is recommended). You can install it in a different ways. If you are using Ubuntu Xenial (16.04LTS)
+then you already have Ruby 2.3 installed. In other cases you can install it with [RVM](https://rvm.io/) or [rbenv](https://github.com/rbenv/rbenv).
+
+Just install the gem by running `gem install proxy_fetcher` in your terminal and run it:
+
+```bash
+proxy_fetcher >> proxies.txt # Will download proxies, validate them and write to file
+```
+
+If you need a list of proxies in JSON then pass `--json` argument to the command:
+
+```bash
+proxy_fetcher --json
+
+# Will print:
+# {"proxies":["https://120.26.206.178:8888","https://119.61.13.242:1080","https://117.40.213.26:1080","https://92.62.72.242:1080",
+# "https://58.20.41.172:1080","https://204.116.192.151:35923","https://190.5.96.58:1080","https://170.250.109.97:35923",
+# "https://121.41.82.99:1080","https://77.53.105.155:35923"]}
+
+```
+
+To get all the possible options run:
+
+```bash
+proxy_fetcher --help
+```
+
+## Proxy object
+
 Every proxy is a `ProxyFetcher::Proxy` object that has next readers (instance variables):
 
 * `addr` (IP address)
@@ -84,7 +148,7 @@ Every proxy is a `ProxyFetcher::Proxy` object that has next readers (instance va
 * `response_time` (5217 for example)
 * `speed` (`:slow`, `:medium` or `:fast`. **Note:** depends on the proxy provider and can be `nil`)
 * `type` (URI schema, HTTP or HTTPS)
-* `anonimity` (Low or High +KA for example)
+* `anonymity` (`Low`, `Elite proxy` or `High +KA` for example)
 
 Also you can call next instance methods for every Proxy object:
 
@@ -94,18 +158,7 @@ Also you can call next instance methods for every Proxy object:
 * `uri` (returns `URI::Generic` object)
 * `url` (returns a formatted URL like "_http://IP:PORT_" )
 
-You can use two methods to get the first proxy from the list:
-
-* `get` or aliased `pop` (will return first proxy and move it to the end of the list)
-* `get!` or aliased `pop!` (will return first **connectable** proxy and move it to the end of the list; all the proxies till the working one will be removed)
-
-If you wanna clear current proxy manager list from dead servers, you can just call `cleanup!` method:
-
-```ruby
-manager.cleanup! # or manager.validate!
-```
-
-You can sort or find any proxy by speed using next 3 instance methods:
+You can sort or find any proxy by speed using next 3 instance methods (if it is available for the specific provider):
 
 * `fast?`
 * `medium?`
@@ -117,26 +170,27 @@ To change open/read timeout for `cleanup!` and `connectable?` methods you need t
 
 ```ruby
 ProxyFetcher.configure do |config|
-  config.read_timeout = 1 # default is 3
-  config.open_timeout = 1 # default is 3
+  config.connection_timeout = 1 # default is 3
 end
 
 manager = ProxyFetcher::Manager.new
 manager.cleanup!
 ```
 
-ProxyFetcher uses simple Ruby solution for dealing with HTTP requests - `net/http` library. If you wanna add, for example, your custom provider that
-was developed as a Single Page Application (SPA) with some JavaScript, then you will need something like []selenium-webdriver](https://github.com/SeleniumHQ/selenium/tree/master/rb)
+ProxyFetcher uses simple Ruby solution for dealing with HTTP(S) requests - `net/http` library from the stdlib. If you wanna add, for example, your custom provider that
+was developed as a Single Page Application (SPA) with some JavaScript, then you will need something like [selenium-webdriver](https://github.com/SeleniumHQ/selenium/tree/master/rb)
 to properly load the content of the website. For those and other cases you can write your own class for fetching HTML content by the URL and setup it
 in the ProxyFetcher config:
 
 ```ruby
 class MyHTTPClient
-  class << self
-    # [IMPORTANT]: self.fetch method is required!
-    def fetch(url)
-      # ... some magic to return proper HTML ...
-    end
+  # [IMPORTANT]: below methods are required!
+  def self.fetch(url)
+    # ... some magic to return proper HTML ...
+  end
+  
+  def self.connectable?(url)
+    # ... some magic to check if url is connectable ...
   end
 end
 
@@ -149,14 +203,17 @@ manager.proxies
  #     @response_time=5217, @speed=48, @type="HTTP", @anonymity="High">, ... ]
 ```
 
+You can take a look at the [lib/proxy_fetcher/utils/http_client.rb](lib/proxy_fetcher/utils/http_client.rb) for an example.
+
 ## Providers
 
 Currently ProxyFetcher can deal with next proxy providers (services):
 
 * Hide My Name (default one)
 * Free Proxy List
-* SSL Proxies
+* Free SSL Proxies
 * Proxy Docker
+* Proxy List
 * XRoxy
 
 If you wanna use one of them just setup required in the config:
@@ -176,14 +233,8 @@ Also you can write your own provider. All you need is to create a class, that wo
 ProxyFetcher::Configuration.register_provider(:your_provider, YourProviderClass)
 ```
 
-Provider class must implement `self.load_proxy_list` and `#parse!(html_entry)` methods that will load and parse
-provider HTML page with proxy list. Take a look at the samples in the `proxy_fetcher/providers` directory.
-
-## TODO
-
-* Add proxy filters
-* Code refactoring
-* Rewrite specs
+Provider class must implement `self.load_proxy_list` and `#to_proxy(html_element)` methods that will load and parse
+provider HTML page with proxy list. Take a look at the existing providers in the [lib/proxy_fetcher/providers](lib/proxy_fetcher/providers) directory.
 
 ## Contributing
 
@@ -206,8 +257,6 @@ Thanks.
 
 ## License
 
-proxy_fetcher gem is released under the [MIT License](http://www.opensource.org/licenses/MIT).
+`proxy_fetcher` gem is released under the [MIT License](http://www.opensource.org/licenses/MIT).
 
 Copyright (c) 2017 Nikita Bulai (bulajnikita@gmail.com).
-
-Some parser code (c) [pifleo](https://gist.github.com/pifleo/3889803)

data/bin/proxy_fetcher

@@ -0,0 +1,57 @@
+#!/usr/bin/env ruby
+
+require 'optparse'
+require 'proxy_fetcher'
+
+options = {
+  validate: true,
+  json: false
+}
+
+OptionParser.new do |opts|
+  opts.banner = 'Usage: proxy_fetcher [OPTIONS]'
+
+  opts.on('-h', '--help', '# Show this help message and quit') do
+    puts opts
+    exit(0)
+  end
+
+  opts.on('-p', '--provider=NAME', '# Use specific proxy provider') do |value|
+    provider_name = value.downcase
+
+    unless ProxyFetcher::Configuration.providers.include?(provider_name.to_sym)
+      possible_providers = ProxyFetcher::Configuration.providers.keys
+
+      puts "Unknown provider - '#{value}'.\nUse one of the following: #{possible_providers.join(', ')}."
+      exit(0)
+    end
+
+    options[:provider] = provider_name
+  end
+
+  opts.on('-n', '--no-validate', '# Dump all the proxies without validation') do
+    options[:validate] = false
+  end
+
+  opts.on('-t', '--timeout=SECONDS', Integer, '# Connection timeout in seconds') do |value|
+    options[:timeout] = value
+  end
+
+  opts.on('-j', '--json', '# Dump proxies to the JSON format') do
+    options[:json] = true
+  end
+end.parse!
+
+ProxyFetcher.config.provider = options[:provider] if options[:provider]
+ProxyFetcher.config.connection_timeout = options[:timeout] if options[:timeout]
+
+manager = ProxyFetcher::Manager.new
+manager.validate! if options[:validate]
+
+if options[:json]
+  require 'json'
+
+  puts JSON.generate(proxies: manager.raw_proxies)
+else
+  puts manager.raw_proxies
+end

data/lib/proxy_fetcher.rb

@@ -1,16 +1,22 @@
 require 'uri'
 require 'net/http'
+require 'openssl'
 require 'nokogiri'
+require 'ostruct'
 
 require 'proxy_fetcher/configuration'
 require 'proxy_fetcher/proxy'
 require 'proxy_fetcher/manager'
-require 'proxy_fetcher/utils/http_fetcher'
+
+require 'proxy_fetcher/utils/http_client'
+require 'proxy_fetcher/utils/html'
+
 require 'proxy_fetcher/providers/base'
 require 'proxy_fetcher/providers/free_proxy_list'
 require 'proxy_fetcher/providers/free_proxy_list_ssl'
 require 'proxy_fetcher/providers/hide_my_name'
 require 'proxy_fetcher/providers/proxy_docker'
+require 'proxy_fetcher/providers/proxy_list'
 require 'proxy_fetcher/providers/xroxy'
 
 module ProxyFetcher

data/lib/proxy_fetcher/configuration.rb

@@ -2,9 +2,10 @@ module ProxyFetcher
   class Configuration
     UnknownProvider = Class.new(StandardError)
     RegisteredProvider = Class.new(StandardError)
+    WrongHttpClient = Class.new(StandardError)
 
-    attr_accessor :open_timeout, :read_timeout, :provider
-    attr_accessor :http_client
+    attr_accessor :http_client, :connection_timeout
+    attr_accessor :provider
 
     class << self
       def providers
@@ -12,15 +13,18 @@ module ProxyFetcher
       end
 
       def register_provider(name, klass)
-        raise RegisteredProvider, "#{name} provider already registered!" if providers.key?(name.to_sym)
+        raise RegisteredProvider, "`#{name}` provider already registered!" if providers.key?(name.to_sym)
 
         providers[name.to_sym] = klass
       end
     end
 
     def initialize
-      @open_timeout = 3
-      @read_timeout = 3
+      reset!
+    end
+
+    def reset!
+      @connection_timeout = 3
       @http_client = HTTPClient
 
       self.provider = :hide_my_name # currently default one
@@ -29,7 +33,15 @@ module ProxyFetcher
     def provider=(name)
       @provider = self.class.providers[name.to_sym]
 
-      raise UnknownProvider, "unregistered proxy provider (#{name})!" if @provider.nil?
+      raise UnknownProvider, "unregistered proxy provider `#{name}`!" if @provider.nil?
+    end
+
+    def http_client=(klass)
+      unless klass.respond_to?(:fetch, :connectable?)
+        raise WrongHttpClient, "#{klass} must respond to #fetch and #connectable? class methods!"
+      end
+
+      @http_client = klass
     end
   end
 end

data/lib/proxy_fetcher/manager.rb

@@ -1,10 +1,12 @@
 module ProxyFetcher
   class Manager
-    attr_reader :proxies
+    attr_reader :proxies, :filters
 
     # refresh: true - load proxy list from the remote server on initialization
     # refresh: false - just initialize the class, proxy list will be empty ([])
-    def initialize(refresh: true)
+    def initialize(refresh: true, filters: {})
+      @filters = filters
+
       if refresh
         refresh_list!
       else
@@ -14,8 +16,7 @@ module ProxyFetcher
 
     # Update current proxy list from the provider
     def refresh_list!
-      rows = ProxyFetcher.config.provider.load_proxy_list
-      @proxies = rows.map { |row| Proxy.new(row) }
+      @proxies = ProxyFetcher.config.provider.fetch_proxies!(filters)
     end
 
     alias fetch! refresh_list!
@@ -56,10 +57,12 @@ module ProxyFetcher
     alias validate! cleanup!
 
     # Return random proxy
-    def random
+    def random_proxy
       proxies.sample
     end
 
+    alias random random_proxy
+
     # Returns array of proxy URLs (just schema + host + port)
     def raw_proxies
       proxies.map(&:url)

data/lib/proxy_fetcher/providers/base.rb

@@ -1,25 +1,52 @@
+require 'forwardable'
+
 module ProxyFetcher
   module Providers
     class Base
-      attr_reader :proxy
+      extend Forwardable
 
-      def initialize(proxy_instance)
-        @proxy = proxy_instance
-      end
+      def_delegators ProxyFetcher::HTML, :clear, :convert_to_int
+
+      PROXY_TYPES = [
+        HTTP = 'HTTP'.freeze,
+        HTTPS = 'HTTPS'.freeze
+      ].freeze
 
-      def set!(name, value)
-        @proxy.instance_variable_set(:"@#{name}", value)
+      attr_reader :proxy
+
+      def fetch_proxies!(filters = {})
+        load_proxy_list(filters).map { |html| to_proxy(html) }
       end
 
       class << self
-        def parse_entry(entry, proxy_instance)
-          new(proxy_instance).parse!(entry)
+        def fetch_proxies!(filters = {})
+          new.fetch_proxies!(filters)
         end
+      end
 
-        # Get HTML from the requested URL
-        def load_html(url)
-          ProxyFetcher.config.http_client.fetch(url)
-        end
+      protected
+
+      # Loads HTML document with Nokogiri by the URL combined with custom filters
+      def load_document(url, filters = {})
+        uri = URI.parse(url)
+        uri.query = URI.encode_www_form(filters) if filters.any?
+
+        Nokogiri::HTML(ProxyFetcher.config.http_client.fetch(uri.to_s))
+      end
+
+      # Get HTML elements with proxy info
+      def load_proxy_list(*)
+        raise NotImplementedError, "#{__method__} must be implemented in a descendant class!"
+      end
+
+      # Convert HTML element with proxy info to ProxyFetcher::Proxy instance
+      def to_proxy(*)
+        raise NotImplementedError, "#{__method__} must be implemented in a descendant class!"
+      end
+
+      # Return normalized HTML element content by selector
+      def parse_element(element, selector, method = :at_xpath)
+        clear(element.public_send(method, selector).content)
       end
     end
   end

data/lib/proxy_fetcher/providers/free_proxy_list.rb

@@ -3,42 +3,27 @@ module ProxyFetcher
     class FreeProxyList < Base
       PROVIDER_URL = 'https://free-proxy-list.net/'.freeze
 
-      class << self
-        def load_proxy_list
-          doc = Nokogiri::HTML(load_html(PROVIDER_URL))
-          doc.xpath('//table[@id="proxylisttable"]/tbody/tr')
-        end
+      # [NOTE] Doesn't support filtering
+      def load_proxy_list(*)
+        doc = load_document(PROVIDER_URL, {})
+        doc.xpath('//table[@id="proxylisttable"]/tbody/tr')
       end
 
-      def parse!(html_entry)
-        html_entry.xpath('td').each_with_index do |td, index|
-          case index
-          when 0
-            set!(:addr, td.content.strip)
-          when 1 then
-            set!(:port, Integer(td.content.strip))
-          when 3 then
-            set!(:country, td.content.strip)
-          when 4
-            set!(:anonymity, td.content.strip)
-          when 6
-            set!(:type, parse_type(td))
-          else
-            # nothing
-          end
+      def to_proxy(html_element)
+        ProxyFetcher::Proxy.new.tap do |proxy|
+          proxy.addr = parse_element(html_element, 'td[1]')
+          proxy.port = convert_to_int(parse_element(html_element, 'td[2]'))
+          proxy.country = parse_element(html_element, 'td[4]')
+          proxy.anonymity = parse_element(html_element, 'td[5]')
+          proxy.type = parse_type(html_element)
         end
       end
 
       private
 
-      def parse_type(td)
-        type = td.content.strip
-
-        if type && type.downcase.include?('yes')
-          'HTTPS'
-        else
-          'HTTP'
-        end
+      def parse_type(element)
+        type = parse_element(element, 'td[6]')
+        type && type.casecmp('yes').zero? ? HTTPS : HTTP
       end
     end
 

data/lib/proxy_fetcher/providers/free_proxy_list_ssl.rb

@@ -3,29 +3,19 @@ module ProxyFetcher
     class FreeProxyListSSL < Base
       PROVIDER_URL = 'https://www.sslproxies.org/'.freeze
 
-      class << self
-        def load_proxy_list
-          doc = Nokogiri::HTML(load_html(PROVIDER_URL))
-          doc.xpath('//table[@id="proxylisttable"]/tbody/tr')
-        end
+      # [NOTE] Doesn't support filtering
+      def load_proxy_list(*)
+        doc = load_document(PROVIDER_URL, {})
+        doc.xpath('//table[@id="proxylisttable"]/tbody/tr')
       end
 
-      def parse!(html_entry)
-        html_entry.xpath('td').each_with_index do |td, index|
-          case index
-          when 0
-            set!(:addr, td.content.strip)
-          when 1 then
-            set!(:port, Integer(td.content.strip))
-          when 3 then
-            set!(:country, td.content.strip)
-          when 4
-            set!(:anonymity, td.content.strip)
-          when 6
-            set!(:type, 'HTTPS')
-          else
-            # nothing
-          end
+      def to_proxy(html_element)
+        ProxyFetcher::Proxy.new.tap do |proxy|
+          proxy.addr = parse_element(html_element, 'td[1]')
+          proxy.port = convert_to_int(parse_element(html_element, 'td[2]'))
+          proxy.country = parse_element(html_element, 'td[4]')
+          proxy.anonymity = parse_element(html_element, 'td[5]')
+          proxy.type = HTTPS
         end
       end
     end

data/lib/proxy_fetcher/providers/hide_my_name.rb

@@ -1,51 +1,49 @@
 module ProxyFetcher
   module Providers
     class HideMyName < Base
-      PROVIDER_URL = 'https://hidemy.name/en/proxy-list/?type=hs'.freeze
+      PROVIDER_URL = 'https://hidemy.name/en/proxy-list/'.freeze
 
-      class << self
-        def load_proxy_list
-          doc = Nokogiri::HTML(load_html(PROVIDER_URL))
-          doc.xpath('//table[@class="proxy__t"]/tbody/tr')
-        end
+      def load_proxy_list(filters = { type: 'hs' })
+        doc = load_document(PROVIDER_URL, filters)
+        doc.xpath('//table[@class="proxy__t"]/tbody/tr')
       end
 
-      def parse!(html_entry)
-        html_entry.xpath('td').each_with_index do |td, index|
-          case index
-          when 0
-            set!(:addr, td.content.strip)
-          when 1 then
-            set!(:port, Integer(td.content.strip))
-          when 2 then
-            set!(:country, td.at_xpath('*//span[1]/following-sibling::text()[1]').content.strip)
-          when 3
-            response_time = Integer(td.at('p').content.strip[/\d+/])
-
-            set!(:response_time, response_time)
-            set!(:speed, speed_from_response_time(response_time))
-          when 4
-            set!(:type, parse_type(td))
-          when 5
-            set!(:anonymity, td.content.strip)
-          else
-            # nothing
-          end
+      def to_proxy(html_element)
+        ProxyFetcher::Proxy.new.tap do |proxy|
+          proxy.addr = parse_element(html_element, 'td[1]')
+          proxy.port = convert_to_int(parse_element(html_element, 'td[2]'))
+          proxy.anonymity = parse_element(html_element, 'td[6]')
+
+          proxy.country = parse_country(html_element)
+          proxy.type = parse_type(html_element)
+
+          response_time = parse_response_time(html_element)
+
+          proxy.response_time = response_time
+          proxy.speed = speed_from_response_time(response_time)
         end
       end
 
       private
 
-      def parse_type(td)
-        schemas = td.content.strip
+      def parse_country(element)
+        clear(element.at_xpath('*//span[1]/following-sibling::text()[1]').content)
+      end
+
+      def parse_type(element)
+        schemas = parse_element(element, 'td[5]')
 
         if schemas && schemas.downcase.include?('https')
-          'HTTPS'
+          HTTPS
         else
-          'HTTP'
+          HTTP
         end
       end
 
+      def parse_response_time(element)
+        convert_to_int(element.at_xpath('td[4]').content.strip[/\d+/])
+      end
+
       def speed_from_response_time(response_time)
         if response_time < 1500
           :fast

data/lib/proxy_fetcher/providers/proxy_docker.rb

@@ -3,30 +3,21 @@ module ProxyFetcher
     class ProxyDocker < Base
       PROVIDER_URL = 'https://www.proxydocker.com/en'.freeze
 
-      class << self
-        def load_proxy_list
-          doc = Nokogiri::HTML(load_html(PROVIDER_URL))
-          doc.xpath('//table[contains(@class, "table")]/tr[(not(@id="proxy-table-header")) and (count(td)>2)]')
-        end
+      # [NOTE] Doesn't support direct filters
+      def load_proxy_list(*)
+        doc = load_document(PROVIDER_URL, {})
+        doc.xpath('//table[contains(@class, "table")]/tr[(not(@id="proxy-table-header")) and (count(td)>2)]')
       end
 
-      def parse!(html_entry)
-        html_entry.xpath('td').each_with_index do |td, index|
-          case index
-          when 0
-            uri = URI("//#{td.content.strip}")
+      def to_proxy(html_element)
+        ProxyFetcher::Proxy.new.tap do |proxy|
+          uri = URI("//#{parse_element(html_element, 'td[1]')}")
+          proxy.addr = uri.host
+          proxy.port = uri.port
 
-            set!(:addr, uri.host)
-            set!(:port, uri.port)
-          when 1
-            set!(:type,  td.content.strip)
-          when 2
-            set!(:anonymity, td.content.strip)
-          when 4 then
-            set!(:country, td.content.strip)
-          else
-            # nothing
-          end
+          proxy.type = parse_element(html_element, 'td[2]')
+          proxy.anonymity = parse_element(html_element, 'td[3]')
+          proxy.country = parse_element(html_element, 'td[5]')
         end
       end
     end

data/lib/proxy_fetcher/providers/proxy_list.rb

@@ -0,0 +1,35 @@
+require 'base64'
+
+module ProxyFetcher
+  module Providers
+    class ProxyList < Base
+      PROVIDER_URL = 'https://proxy-list.org/english/index.php'.freeze
+
+      def load_proxy_list(filters = {})
+        doc = load_document(PROVIDER_URL, filters)
+        doc.css('.table-wrap .table ul')
+      end
+
+      def to_proxy(html_element)
+        ProxyFetcher::Proxy.new.tap do |proxy|
+          uri = parse_proxy_uri(html_element)
+          proxy.addr = uri.host
+          proxy.port = uri.port
+
+          proxy.type = parse_element(html_element, 'li[2]')
+          proxy.anonymity = parse_element(html_element, 'li[4]')
+          proxy.country = clear(html_element.at_xpath("li[5]//span[@class='country']").attr('title'))
+        end
+      end
+
+      private
+
+      def parse_proxy_uri(element)
+        full_addr = ::Base64.decode64(element.at('li script').inner_html.match(/'(.+)'/)[1])
+        URI.parse("http://#{full_addr}")
+      end
+    end
+
+    ProxyFetcher::Configuration.register_provider(:proxy_list, ProxyList)
+  end
+end

data/lib/proxy_fetcher/providers/xroxy.rb

@@ -1,34 +1,21 @@
 module ProxyFetcher
   module Providers
     class XRoxy < Base
-      PROVIDER_URL = 'http://www.xroxy.com/proxylist.php?port=&type=All_http'.freeze
+      PROVIDER_URL = 'http://www.xroxy.com/proxylist.php'.freeze
 
-      class << self
-        def load_proxy_list
-          doc = Nokogiri::HTML(load_html(PROVIDER_URL))
-          doc.xpath('//div[@id="content"]/table[1]/tr[contains(@class, "row")]')
-        end
+      def load_proxy_list(filters = { type: 'All_http' })
+        doc = load_document(PROVIDER_URL, filters)
+        doc.xpath('//div[@id="content"]/table[1]/tr[contains(@class, "row")]')
       end
 
-      def parse!(html_entry)
-        html_entry.xpath('td').each_with_index do |td, index|
-          case index
-          when 1
-            set!(:addr, td.content.strip)
-          when 2
-            set!(:port, Integer(td.content.strip))
-          when 3
-            set!(:anonymity,  td.content.strip)
-          when 4
-            ssl = td.content.strip.downcase
-            set!(:type, ssl.include?('true') ? 'HTTPS' : 'HTTP')
-          when 5 then
-            set!(:country, td.content.strip)
-          when 6
-            set!(:response_time, Integer(td.content.strip))
-          else
-            # nothing
-          end
+      def to_proxy(html_element)
+        ProxyFetcher::Proxy.new.tap do |proxy|
+          proxy.addr = parse_element(html_element, 'td[2]')
+          proxy.port = convert_to_int(parse_element(html_element, 'td[3]'))
+          proxy.anonymity = parse_element(html_element, 'td[4]')
+          proxy.type = parse_element(html_element, 'td[5]').casecmp('true').zero? ? HTTPS : HTTP
+          proxy.country = parse_element(html_element, 'td[6]')
+          proxy.response_time = convert_to_int(parse_element(html_element, 'td[7]'))
         end
       end
     end

data/lib/proxy_fetcher/proxy.rb

@@ -1,24 +1,7 @@
 module ProxyFetcher
-  class Proxy
-    attr_reader :addr, :port, :country, :response_time, :speed, :type, :anonymity
-
-    def initialize(html_row)
-      ProxyFetcher.config.provider.parse_entry(html_row, self)
-
-      self
-    end
-
+  class Proxy < OpenStruct
     def connectable?
-      connection = Net::HTTP.new(addr, port)
-      connection.use_ssl = true if https?
-      connection.open_timeout = ProxyFetcher.config.open_timeout
-      connection.read_timeout = ProxyFetcher.config.read_timeout
-
-      connection.start { |http| return true if http.request_head('/') }
-
-      false
-    rescue Timeout::Error, Errno::ECONNREFUSED, Errno::ECONNRESET, Errno::ECONNABORTED
-      false
+      ProxyFetcher.config.http_client.connectable?(url)
     end
 
     alias valid? connectable?

data/lib/proxy_fetcher/utils/html.rb

@@ -0,0 +1,15 @@
+module ProxyFetcher
+  class HTML
+    class << self
+      def clear(text)
+        return if text.nil? || text.empty?
+
+        text.strip.gsub(/[ \t]/i, '')
+      end
+
+      def convert_to_int(text)
+        Integer(clear(text))
+      end
+    end
+  end
+end

data/lib/proxy_fetcher/utils/http_client.rb

@@ -0,0 +1,46 @@
+module ProxyFetcher
+  class HTTPClient
+    attr_reader :uri, :http
+
+    def initialize(url)
+      @uri = URI.parse(url)
+      @http = Net::HTTP.new(@uri.host, @uri.port)
+      return unless https?
+
+      @http.use_ssl = true
+      @http.verify_mode = OpenSSL::SSL::VERIFY_NONE
+    end
+
+    def fetch
+      request = Net::HTTP::Get.new(@uri.to_s)
+      request['Connection'] = 'keep-alive'
+      response = @http.request(request)
+      response.body
+    end
+
+    def connectable?
+      @http.open_timeout = ProxyFetcher.config.connection_timeout
+      @http.read_timeout = ProxyFetcher.config.connection_timeout
+
+      @http.start { |connection| return true if connection.request_head('/') }
+
+      false
+    rescue StandardError
+      false
+    end
+
+    def https?
+      @uri.scheme.casecmp('https').zero?
+    end
+
+    class << self
+      def fetch(url)
+        new(url).fetch
+      end
+
+      def connectable?(url)
+        new(url).connectable?
+      end
+    end
+  end
+end

data/lib/proxy_fetcher/version.rb

@@ -7,9 +7,9 @@ module ProxyFetcher
     # Major version number
     MAJOR = 0
     # Minor version number
-    MINOR = 2
+    MINOR = 3
     # Smallest version number
-    TINY  = 5
+    TINY  = 0
 
     # Full version number
     STRING = [MAJOR, MINOR, TINY].compact.join('.')

data/proxy_fetcher.gemspec

@@ -5,14 +5,16 @@ require 'proxy_fetcher/version'
 Gem::Specification.new do |gem|
   gem.name = 'proxy_fetcher'
   gem.version = ProxyFetcher.gem_version
-  gem.date = '2017-08-17'
-  gem.summary = 'Ruby gem for dealing with proxy lists '
+  gem.date = '2017-08-21'
+  gem.summary = 'Ruby gem for dealing with proxy lists from different providers'
   gem.description = 'This gem can help your Ruby application to make HTTP(S) requests ' \
                     'from proxy server by fetching and validating proxy lists from the different providers.'
   gem.authors = ['Nikita Bulai']
   gem.email = 'bulajnikita@gmail.com'
   gem.require_paths = ['lib']
+  gem.bindir = 'bin'
   gem.files = `git ls-files`.split($RS)
+  gem.executables = `git ls-files -- bin/*`.split("\n").map { |f| File.basename(f) }
   gem.homepage = 'http://github.com/nbulaj/proxy_fetcher'
   gem.license = 'MIT'
   gem.required_ruby_version = '>= 2.2.2'

data/spec/proxy_fetcher/configuration_spec.rb

@@ -0,0 +1,48 @@
+require 'spec_helper'
+
+describe ProxyFetcher::Configuration do
+  before { ProxyFetcher.config.reset! }
+  after { ProxyFetcher.config.reset! }
+
+  context 'custom HTTP client' do
+    it 'successfully setups if class has all the required methods' do
+      class MyHTTPClient
+        def self.fetch(url)
+          url
+        end
+
+        def self.connectable?(*)
+          true
+        end
+      end
+
+      expect { ProxyFetcher.config.http_client = MyHTTPClient }.not_to raise_error
+    end
+
+    it 'failed on setup if required methods are missing' do
+      MyWrongHTTPClient = Class.new
+
+      expect { ProxyFetcher.config.http_client = MyWrongHTTPClient }
+        .to raise_error(ProxyFetcher::Configuration::WrongHttpClient)
+    end
+  end
+
+  context 'custom provider' do
+    it 'successfully setups if provider class registered' do
+      CustomProvider = Class.new(ProxyFetcher::Providers::Base)
+      ProxyFetcher::Configuration.register_provider(:custom_provider, CustomProvider)
+
+      expect { ProxyFetcher.config.provider = :custom_provider }.not_to raise_error
+    end
+
+    it 'failed on setup if provider class is not registered' do
+      expect { ProxyFetcher.config.provider = :unexisting_provider }
+        .to raise_error(ProxyFetcher::Configuration::UnknownProvider)
+    end
+
+    it 'failed on setup if provider class already registered' do
+      expect { ProxyFetcher::Configuration.register_provider(:xroxy, Class.new)}
+        .to raise_error(ProxyFetcher::Configuration::RegisteredProvider)
+    end
+  end
+end

data/spec/proxy_fetcher/providers/base_spec.rb

@@ -0,0 +1,28 @@
+require 'spec_helper'
+
+describe ProxyFetcher::Providers::Base do
+  before { ProxyFetcher.config.reset! }
+  after { ProxyFetcher.config.reset! }
+
+  it 'does not allows to use not implemented methods' do
+    NotImplementedCustomProvider = Class.new(ProxyFetcher::Providers::Base)
+
+    ProxyFetcher::Configuration.register_provider(:provider_without_methods, NotImplementedCustomProvider)
+    ProxyFetcher.config.provider = :provider_without_methods
+
+    expect { ProxyFetcher::Manager.new }.to raise_error(NotImplementedError) do |error|
+      expect(error.message).to include('load_proxy_list')
+    end
+
+    # implement one of the methods
+    NotImplementedCustomProvider.class_eval do
+      def load_proxy_list(*)
+        [1, 2, 3]
+      end
+    end
+
+    expect { ProxyFetcher::Manager.new }.to raise_error(NotImplementedError) do |error|
+      expect(error.message).to include('to_proxy')
+    end
+  end
+end

data/spec/proxy_fetcher/providers/proxy_list_spec.rb

@@ -0,0 +1,9 @@
+require 'spec_helper'
+
+describe ProxyFetcher::Providers::ProxyList do
+  before :all do
+    ProxyFetcher.config.provider = :proxy_list
+  end
+
+  it_behaves_like 'a manager'
+end

data/spec/proxy_fetcher/proxy_spec.rb

@@ -9,24 +9,24 @@ describe ProxyFetcher::Proxy do
     @manager = ProxyFetcher::Manager.new
   end
 
-  let(:proxy) { @manager.proxies.first }
+  let(:proxy) { @manager.proxies.first.dup }
 
   it 'checks schema' do
-    proxy.instance_variable_set(:@type, 'HTTP')
+    proxy.type = ProxyFetcher::Providers::Base::HTTP
     expect(proxy.http?).to be_truthy
     expect(proxy.https?).to be_falsey
 
-    proxy.instance_variable_set(:@type, 'HTTPS')
+    proxy.type = ProxyFetcher::Providers::Base::HTTPS
     expect(proxy.https?).to be_truthy
     expect(proxy.http?).to be_falsey
   end
 
   it 'not connectable if IP addr is wrong' do
-    allow_any_instance_of(ProxyFetcher::Proxy).to receive(:addr).and_return('192.168.1.1')
+    proxy.addr = '192.168.1.0'
     expect(proxy.connectable?).to be_falsey
   end
 
-  it 'not connectable if ERR' do
+  it 'not connectable if there are some error during connection request' do
     allow_any_instance_of(Net::HTTP).to receive(:start).and_raise(Errno::ECONNABORTED)
     expect(proxy.connectable?).to be_falsey
   end
@@ -46,13 +46,13 @@ describe ProxyFetcher::Proxy do
   end
 
   it 'checks speed' do
-    proxy.instance_variable_set(:@speed, :fast)
+    proxy.speed = :fast
     expect(proxy.fast?).to be_truthy
 
-    proxy.instance_variable_set(:@speed, :slow)
+    proxy.speed = :slow
     expect(proxy.slow?).to be_truthy
 
-    proxy.instance_variable_set(:@speed, :medium)
+    proxy.speed = :medium
     expect(proxy.medium?).to be_truthy
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: proxy_fetcher
 version: !ruby/object:Gem::Version
-  version: 0.2.5
+  version: 0.3.0
 platform: ruby
 authors:
 - Nikita Bulai
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-08-17 00:00:00.000000000 Z
+date: 2017-08-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: nokogiri
@@ -47,7 +47,8 @@ dependencies:
 description: This gem can help your Ruby application to make HTTP(S) requests from
   proxy server by fetching and validating proxy lists from the different providers.
 email: bulajnikita@gmail.com
-executables: []
+executables:
+- proxy_fetcher
 extensions: []
 extra_rdoc_files: []
 files:
@@ -58,6 +59,7 @@ files:
 - LICENSE
 - README.md
 - Rakefile
+- bin/proxy_fetcher
 - lib/proxy_fetcher.rb
 - lib/proxy_fetcher/configuration.rb
 - lib/proxy_fetcher/manager.rb
@@ -66,15 +68,20 @@ files:
 - lib/proxy_fetcher/providers/free_proxy_list_ssl.rb
 - lib/proxy_fetcher/providers/hide_my_name.rb
 - lib/proxy_fetcher/providers/proxy_docker.rb
+- lib/proxy_fetcher/providers/proxy_list.rb
 - lib/proxy_fetcher/providers/xroxy.rb
 - lib/proxy_fetcher/proxy.rb
-- lib/proxy_fetcher/utils/http_fetcher.rb
+- lib/proxy_fetcher/utils/html.rb
+- lib/proxy_fetcher/utils/http_client.rb
 - lib/proxy_fetcher/version.rb
 - proxy_fetcher.gemspec
+- spec/proxy_fetcher/configuration_spec.rb
+- spec/proxy_fetcher/providers/base_spec.rb
 - spec/proxy_fetcher/providers/free_proxy_list_spec.rb
 - spec/proxy_fetcher/providers/free_proxy_list_ssl_spec.rb
 - spec/proxy_fetcher/providers/hide_my_name_spec.rb
 - spec/proxy_fetcher/providers/proxy_docker_spec.rb
+- spec/proxy_fetcher/providers/proxy_list_spec.rb
 - spec/proxy_fetcher/providers/xroxy_spec.rb
 - spec/proxy_fetcher/proxy_spec.rb
 - spec/spec_helper.rb
@@ -102,5 +109,5 @@ rubyforge_project:
 rubygems_version: 2.6.11
 signing_key: 
 specification_version: 4
-summary: Ruby gem for dealing with proxy lists
+summary: Ruby gem for dealing with proxy lists from different providers
 test_files: []

data/lib/proxy_fetcher/utils/http_fetcher.rb

@@ -1,24 +0,0 @@
-module ProxyFetcher
-  class HTTPClient
-    attr_reader :http
-
-    def initialize(url)
-      @uri = URI.parse(url)
-      @http = Net::HTTP.new(@uri.host, @uri.port)
-      @http.use_ssl = true if @uri.scheme.downcase == 'https'
-    end
-
-    def fetch
-      request = Net::HTTP::Get.new(@uri.to_s)
-      request['Connection'] = 'keep-alive'
-      response = @http.request(request)
-      response.body
-    end
-
-    class << self
-      def fetch(url)
-        new(url).fetch
-      end
-    end
-  end
-end