proxy_fetcher 0.3.1 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 27d97dfb3997e004b2fb1a07e935ba7a1132148d
-  data.tar.gz: 8084082dc54f59c2c6bc8c0457b2ea766c479db0
+  metadata.gz: '0529f88d50000f7c2d3fad641372f5dcddfd40b3'
+  data.tar.gz: eab7966ac9aacf6cb62a8673280beed0fbb34330
 SHA512:
-  metadata.gz: 44a592c9882b108ff9f78cc274c27d6e33e8460aa8b3599a0388abb75e787829565223fd4bdfe48a798ca8e153c79c9a3bd1db6184540bf47567b85fa5bf01a9
-  data.tar.gz: 6bce6a4e8d0c197852edd8766c62a2e5850ab75e657f56b90f521307a98a3fa3995efd1219ff963648ad31dad2b5bc467a4c783887b724925e826393605cf36f
+  metadata.gz: 45964603f17dc94e09075a1fb2264bc35a56a0667bcbc03a2142156414a23f35ee2d98aad1326fd64c80be7db8a31f5103244852253b56f50c168bb33fac94c1
+  data.tar.gz: 88ef9ef4e52e31277d0881f0b14b8ebf3095af18b993bee6cf1f0941edd56251a5ee3ea71f7b8ce2f725a127c8c6ef6f9d2c1b5c290103d43879b621cdd1813b

data/.rubocop.yml

@@ -3,6 +3,7 @@ LineLength:
 AllCops:
   Exclude:
     - 'spec/**/*'
+    - 'bin/*'
   DisplayCopNames: true
 Rails:
   Enabled: false

data/CHANGELOG.md

@@ -0,0 +1,83 @@
+# Proxy Fetcher Changelog
+
+Reverse Chronological Order:
+
+## `0.4.0` (2017-08-26)
+
+* Support operations with multiple providers
+* Refactor filtering
+* Small bugfixes
+* Documentation
+
+## `0.3.1` (2017-08-24)
+
+* Remove speed from proxy (no need to)
+* Extract proxy validation from the HTTPClient to separate class
+* Make proxy validator configurable
+* Refactor proxy validation behavior
+* Refactor Proxy object (OpenStruct => PORO, url / uri methods, etc)
+* Optimize proxy list check with threads
+* Improve proxy_fetcher bin
+
+## `0.3.0` (2017-08-21)
+
+* Proxy providers refactoring
+* Proxy object refactoring
+* Specs refactoring
+* New providers
+* Custom HTTP client
+* Configuration improvements
+* Proxy filters
+
+## `0.2.5` (2017-08-17)
+
+* Configurable HTTPClient
+* Fix errors handling
+
+## `0.2.3` (2017-08-10)
+
+* Fix broken providers
+* Add new providers
+* Docs
+
+## `0.2.2` (2017-07-20)
+
+* Code & specs refactoring
+
+## `0.2.1` (2017-07-19)
+
+* New proxy providers
+* Bugfixes
+
+## `0.2.0` (2017-07-17)
+
+* New proxy providers
+* Custom providers
+* Network errors handling
+* Specs refactorirng
+
+## `0.1.4` (2017-05-31)
+
+* Code & specs refactoring
+* Add `speed` to `Proxy` object
+* Docs
+
+## `0.1.3` (2017-05-25)
+
+* Proxy list management with `ProxyFetcher::Manager`
+
+## `0.1.2` (2017-05-23)
+
+* HTTPS proccesing
+* `Proxy` object sugar
+* Specs improvements
+* Docs improvements
+
+## `0.1.1` (2017-05-22)
+
+* Configuration (timeouts)
+* Documentation
+
+## `0.1.0` (2017-05-19)
+
+* Initial release

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,46 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at bulajnikita@gmail.com. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/4/

data/README.md

@@ -6,20 +6,33 @@
 [![License](http://img.shields.io/badge/license-MIT-brightgreen.svg)](#license)
 
 This gem can help your Ruby application to make HTTP(S) requests from proxy by fetching and validating actual
-proxy lists from the different providers like [HideMyName](https://hidemy.name/en/).
+proxy lists from multiple providers like [HideMyName](https://hidemy.name/en/).
 
-It gives you a `Manager` class that can load proxy list, validate it and return random or specific proxy entry. Take a look
+It gives you a `Manager` class that can load proxy lists, validate them and return random or specific proxies. Take a look
 at the documentation below to find all the gem features.
 
-Also this gem can be used as standalone solution for downloading and validating proxy lists from the different providers.
-Checkout examples of usage below.
+Also this gem can be used with any other programming language (Go / Python / etc) as standalone solution for downloading and
+validating proxy lists from the different providers. [Checkout examples](#standalone) of usage below.
+
+## Table of Contents
+
+- [Installation](#installation)
+- [Example of usage](#example-of-usage)
+  - [In Ruby application](#in-ruby-application)
+  - [Standalone](#standalone)
+- [Configuration](#configuration)
+  - [Proxy validation speed](#proxy-validation-speed)
+- [Proxy object](#proxy-object)
+- [Providers](#providers)
+- [Contributing](#contributing)
+- [License](#license)
 
 ## Installation
 
 If using bundler, first add 'proxy_fetcher' to your Gemfile:
 
 ```ruby
-gem 'proxy_fetcher', '~> 0.3'
+gem 'proxy_fetcher', '~> 0.4'
 ```
 
 or if you want to use the latest version (from `master` branch), then:
@@ -37,7 +50,7 @@ bundle install
 Otherwise simply install the gem:
 
 ```sh
-gem install proxy_fetcher -v '0.3'
+gem install proxy_fetcher -v '0.4'
 ```
 
 ## Example of usage
@@ -63,12 +76,17 @@ manager.proxies
  #=> []
 ```
 
-If you wanna clean current proxy list from some dead servers that does not respond to the requests, than you can just call `cleanup!` method:
+If you want to clean current proxy list from the dead servers that does not respond to the requests, than you can just call `cleanup!` method:
 
 ```ruby
 manager.cleanup! # or manager.validate!
 ```
 
+In order to increase the speed of this operation proxy list validation is performed using Ruby threads.
+By default, gem creates a pool with 10 threads, but you can increase this number by passing threads pool
+size to the `#cleanup!` (or `#validate!`) method: `manager.validate!(50)`. In that case ProxyFetcher will
+process all the fetched proxies in group of 50 threads.
+
 Get raw proxy URLs as Strings:
 
 ```ruby
@@ -88,33 +106,61 @@ manager.refresh_list! # or manager.fetch!
  #     @response_time=5217, @type="HTTP", @anonymity="High">, ... ]
 ```
 
-If you need to filter proxy list, for example, by country or response time and selected provider supports filtering by GET params, then you
-can pass your filters to the Manager instance like that:
+You can use two methods to get the first proxy from the list:
+
+* `get` or aliased `pop` (will return first proxy and move it to the end of the list)
+* `get!` or aliased `pop!` (will return first **connectable** proxy and move it to the end of the list; all the proxies till the working one will be removed)
+
+Or you can get just random proxy by calling `manager.random_proxy` or it's alias `manager.random`.
+
+If you need to filter proxy list, for example, by country or response time and selected provider supports filtering with GET params,
+then you can just pass your filters like a simple Ruby hash to the Manager instance:
 
 ```ruby
-ProxyFetcher.config.provider = :hide_my_name
+ProxyFetcher.config.providers = :hide_my_name
 
-manager = ProxyFetcher::Manager.new(filters: { country: 'AO', maxtime: '500' })
+manager = ProxyFetcher::Manager.new(filters: { country: 'PL', maxtime: '500' })
 manager.proxies
 
  # => [...]
 ```
 
-*NOTE*: not all the providers support filtering. Take a look at the provider class to see if it supports custom filters.
+If you are using multiple providers, then you can split your filters by proxy provider names:
 
-You can use two methods to get the first proxy from the list:
+```ruby
+ProxyFetcher.config.providers = [:hide_my_name, :xroxy]
+
+manager = ProxyFetcher::Manager.new(filters: {
+  hide_my_name: {
+    country: 'PL',
+    maxtime: '500'
+  },
+  xroxy: {
+    type: 'All_http'
+  }
+})
+  
+manager.proxies
 
-* `get` or aliased `pop` (will return first proxy and move it to the end of the list)
-* `get!` or aliased `pop!` (will return first **connectable** proxy and move it to the end of the list; all the proxies till the working one will be removed)
+ # => [...]
+```
 
-Or you can get just random proxy by calling `manager.random_proxy` or it's alias `manager.random`.
+You can apply different filters every time you calling `#refresh_list!` (or `#fetch!`) method:
+
+```ruby
+manager.refresh_list!(country: 'PL', maxtime: '500')
+ 
+ # => [...]
+```
+
+*NOTE*: not all the providers support filtering. Take a look at the provider classes to see if it supports custom filters.
 
 ### Standalone
 
-All you need to use this gem is Ruby >= 2.0 (2.3 is recommended). You can install it in a different ways. If you are using Ubuntu Xenial (16.04LTS)
+All you need to use this gem is Ruby >= 2.0 (2.4 is recommended). You can install it in a different ways. If you are using Ubuntu Xenial (16.04LTS)
 then you already have Ruby 2.3 installed. In other cases you can install it with [RVM](https://rvm.io/) or [rbenv](https://github.com/rbenv/rbenv).
 
-Just install the gem by running `gem install proxy_fetcher` in your terminal and run it:
+After installing Ruby just bundle the gem by running `gem install proxy_fetcher` in your terminal and now you can run it:
 
 ```bash
 proxy_fetcher >> proxies.txt # Will download proxies from the default provider, validate them and write to file
@@ -142,27 +188,6 @@ To get all the possible options run:
 proxy_fetcher --help
 ```
 
-## Proxy object
-
-Every proxy is a `ProxyFetcher::Proxy` object that has next readers (instance variables):
-
-* `addr` (IP address)
-* `port`
-* `type` (proxy type, can be HTTP, HTTPS, SOCKS4 or/and SOCKS5)
-* `country` (USA or Brazil for example)
-* `response_time` (5217 for example)
-* `anonymity` (`Low`, `Elite proxy` or `High +KA` for example)
-
-Also you can call next instance methods for every Proxy object:
-
-* `connectable?` (whether proxy server is available)
-* `http?` (whether proxy server has a HTTP protocol)
-* `https?` (whether proxy server has a HTTPS protocol)
-* `socks4?`
-* `socks5?`
-* `uri` (returns `URI::Generic` object)
-* `url` (returns a formatted URL like "_http://IP:PORT_" )
-
 ## Configuration
 
 To change open/read timeout for `cleanup!` and `connectable?` methods you need to change ProxyFetcher.config:
@@ -215,7 +240,7 @@ ProxyFetcher.config.proxy_validator = MyProxyValidator
 manager = ProxyFetcher::Manager.new
 manager.proxies
 
-#=> [#<ProxyFetcher::Proxy:0x00000002879680 @addr="97.77.104.22", @port=3128, @country="USA", 
+ #=> [#<ProxyFetcher::Proxy:0x00000002879680 @addr="97.77.104.22", @port=3128, @country="USA", 
  #     @response_time=5217, @type="HTTP", @anonymity="High">, ... ]
  
 manager.validate!
@@ -223,6 +248,48 @@ manager.validate!
  #=> [ ... ]
 ```
 
+### Proxy validation speed
+
+There are some tricks to increase proxy list validation performance.
+
+In a few words, ProxyFetcher gem uses threads to validate proxies for availability. Every proxy is checked in a
+separate thread. By default, ProxyFetcher uses a pool with a maximum of 10 threads. You can increase this number by
+setting max number of threads in the config:
+
+```ruby
+ProxyFetcher.config.pool_size = 50
+```
+
+You can experiment with the threads pool size to find an optimal number of maximum threads count for you PC and OS.
+This will definitely give you some performance improvements.
+
+Moreover, the common proxy validation speed depends on `ProxyFetcher.config.connection_timeout` option that is equal
+to `3` by default. It means that gem will wait 3 seconds for the server answer to check if particular proxy is connectable.
+You can decrease this option to `1`, for example, and it will heavily increase proxy validation speed (**but remember**
+that some proxies could be connectable, but slow, so with this option you will clear proxy list from the proxies that
+works, but very slow).
+
+## Proxy object
+
+Every proxy is a `ProxyFetcher::Proxy` object that has next readers (instance variables):
+
+* `addr` (IP address)
+* `port`
+* `type` (proxy type, can be HTTP, HTTPS, SOCKS4 or/and SOCKS5)
+* `country` (USA or Brazil for example)
+* `response_time` (5217 for example)
+* `anonymity` (`Low`, `Elite proxy` or `High +KA` for example)
+
+Also you can call next instance methods for every Proxy object:
+
+* `connectable?` (whether proxy server is available)
+* `http?` (whether proxy server has a HTTP protocol)
+* `https?` (whether proxy server has a HTTPS protocol)
+* `socks4?`
+* `socks5?`
+* `uri` (returns `URI::Generic` object)
+* `url` (returns a formatted URL like "_http://IP:PORT_" )
+
 ## Providers
 
 Currently ProxyFetcher can deal with next proxy providers (services):
@@ -234,7 +301,7 @@ Currently ProxyFetcher can deal with next proxy providers (services):
 * Proxy List
 * XRoxy
 
-If you wanna use one of them just setup required in the config:
+If you wanna use one of them just setup it in the config:
 
 ```ruby
 ProxyFetcher.config.provider = :free_proxy_list
@@ -244,7 +311,29 @@ manager.proxies
  #=> ...
 ```
 
-Also you can write your own provider. All you need is to create a class, that would be inherited from the
+You can use multiple providers at the same time:
+
+```ruby
+ProxyFetcher.config.providers = :free_proxy_list, :xroxy, :proxy_docker
+
+manager = ProxyFetcher::Manager.new
+manager.proxies
+ #=> ...
+```
+
+If you want to use all the possible proxy providers then you can configure ProxyFetcher as follows:
+
+```ruby
+ProxyFetcher.config.providers = ProxyFetcher::Configuration.registered_providers
+
+manager = ProxyFetcher::Manager.new.proxies
+manager.proxies
+
+ #=> [#<ProxyFetcher::Proxy:0x00000002879680 @addr="97.77.104.22", @port=3128, @country="USA", 
+ #     @response_time=5217, @type="HTTP", @anonymity="High">, ... ]
+```
+
+Moreover, you can write your own provider! All you need is to create a class, that would be inherited from the
 `ProxyFetcher::Providers::Base` class, and register your provider like this:
 
 ```ruby

data/bin/proxy_fetcher

@@ -17,17 +17,8 @@ OptionParser.new do |opts|
     exit(0)
   end
 
-  opts.on('-p', '--provider=NAME', '# Use specific proxy provider') do |value|
-    provider_name = value.downcase
-
-    unless ProxyFetcher::Configuration.providers.include?(provider_name.to_sym)
-      possible_providers = ProxyFetcher::Configuration.providers.keys
-
-      puts "Unknown provider - '#{value}'.\nUse one of the following: #{possible_providers.join(', ')}."
-      exit(0)
-    end
-
-    options[:provider] = provider_name
+  opts.on('-p', '--providers=NAME1,NAME2', Array, '# Use specific proxy providers') do |values|
+    options[:providers] = values
   end
 
   opts.on('-n', '--no-validate', '# Dump all the proxies without validation') do
@@ -49,7 +40,7 @@ OptionParser.new do |opts|
   end
 end.parse!
 
-ProxyFetcher.config.provider = options[:provider] if options[:provider]
+ProxyFetcher.config.providers = options[:providers] if options[:providers]
 ProxyFetcher.config.connection_timeout = options[:timeout] if options[:timeout]
 
 manager = ProxyFetcher::Manager.new(filters: options[:filters])

data/lib/proxy_fetcher.rb

@@ -4,6 +4,7 @@ require 'nokogiri'
 require 'thread'
 
 require File.dirname(__FILE__) + '/proxy_fetcher/configuration'
+require File.dirname(__FILE__) + '/proxy_fetcher/configuration/providers_registry'
 require File.dirname(__FILE__) + '/proxy_fetcher/proxy'
 require File.dirname(__FILE__) + '/proxy_fetcher/manager'
 

data/lib/proxy_fetcher/configuration.rb

@@ -1,21 +1,21 @@
 module ProxyFetcher
   class Configuration
-    UnknownProvider = Class.new(StandardError)
-    RegisteredProvider = Class.new(StandardError)
     WrongCustomClass = Class.new(StandardError)
 
-    attr_accessor :provider, :connection_timeout
-    attr_accessor :http_client, :proxy_validator, :logger
+    attr_accessor :providers, :connection_timeout, :pool_size
+    attr_accessor :http_client, :proxy_validator
 
     class << self
-      def providers
-        @providers ||= {}
+      def providers_registry
+        @registry ||= ProvidersRegistry.new
       end
 
       def register_provider(name, klass)
-        raise RegisteredProvider, "`#{name}` provider already registered!" if providers.key?(name.to_sym)
+        providers_registry.register(name, klass)
+      end
 
-        providers[name.to_sym] = klass
+      def registered_providers
+        providers_registry.providers.keys
       end
     end
 
@@ -23,20 +23,23 @@ module ProxyFetcher
       reset!
     end
 
+    # Sets default configuration options
     def reset!
+      @pool_size = 10
       @connection_timeout = 3
       @http_client = HTTPClient
       @proxy_validator = ProxyValidator
 
-      self.provider = :hide_my_name # currently default one
+      self.providers = [:hide_my_name] # currently default one
     end
 
-    def provider=(name)
-      @provider = self.class.providers[name.to_sym]
-
-      raise UnknownProvider, "unregistered proxy provider `#{name}`!" if @provider.nil?
+    def providers=(value)
+      @providers = Array(value)
     end
 
+    alias provider providers
+    alias provider= providers=
+
     def http_client=(klass)
       @http_client = setup_custom_class(klass, required_methods: :fetch)
     end
@@ -47,6 +50,7 @@ module ProxyFetcher
 
     private
 
+    # Checks if custom class has some required class methods
     def setup_custom_class(klass, required_methods: [])
       unless klass.respond_to?(*required_methods)
         raise WrongCustomClass, "#{klass} must respond to [#{Array(required_methods).join(', ')}] class methods!"

data/lib/proxy_fetcher/configuration/providers_registry.rb

@@ -0,0 +1,29 @@
+module ProxyFetcher
+  class ProvidersRegistry
+    UnknownProvider = Class.new(StandardError)
+    RegisteredProvider = Class.new(StandardError)
+
+    def providers
+      @providers ||= {}
+    end
+
+    # Add custom provider to common registry.
+    # Requires proxy provider name ('hide_my_name' for example) and a class
+    # that implements the parsing logic.
+    def register(name, klass)
+      raise RegisteredProvider, "`#{name}` provider already registered!" if providers.key?(name.to_sym)
+
+      providers[name.to_sym] = klass
+    end
+
+    # Returns a class for specific provider if it is
+    # registered in the registry. Otherwise throws an exception.
+    def class_for(provider_name)
+      provider_name = provider_name.to_sym
+
+      providers.fetch(provider_name)
+    rescue KeyError
+      raise UnknownProvider, "unregistered proxy provider `#{provider_name}`"
+    end
+  end
+end

data/lib/proxy_fetcher/manager.rb

@@ -1,22 +1,29 @@
 module ProxyFetcher
   class Manager
-    attr_reader :proxies, :filters
+    attr_reader :proxies
 
     # refresh: true - load proxy list from the remote server on initialization
     # refresh: false - just initialize the class, proxy list will be empty ([])
-    def initialize(refresh: true, filters: {})
-      @filters = filters
-
+    def initialize(refresh: true, validate: false, filters: {})
       if refresh
-        refresh_list!
+        refresh_list!(filters)
       else
         @proxies = []
       end
+
+      cleanup! if validate
     end
 
     # Update current proxy list from the provider
-    def refresh_list!
-      @proxies = ProxyFetcher.config.provider.fetch_proxies!(filters)
+    def refresh_list!(filters = nil)
+      @proxies = []
+
+      ProxyFetcher.config.providers.each do |provider_name|
+        provider = ProxyFetcher::Configuration.providers_registry.class_for(provider_name)
+        provider_filters = filters && filters.fetch(provider_name.to_sym, filters)
+
+        @proxies.concat(provider.fetch_proxies!(provider_filters))
+      end
     end
 
     alias fetch! refresh_list!
@@ -50,10 +57,10 @@ module ProxyFetcher
     alias pop! get!
 
     # Clean current proxy list from dead proxies (that doesn't respond by timeout)
-    def cleanup!(pool_size = 10)
+    def cleanup!
       lock = Mutex.new
 
-      proxies.dup.each_slice(pool_size) do |proxy_group|
+      proxies.dup.each_slice(ProxyFetcher.config.pool_size) do |proxy_group|
         threads = proxy_group.map do |group_proxy|
           Thread.new(group_proxy, proxies) do |proxy, proxies|
             lock.synchronize { proxies.delete(proxy) } unless proxy.connectable?

data/lib/proxy_fetcher/providers/base.rb

@@ -7,8 +7,8 @@ module ProxyFetcher
 
       def_delegators ProxyFetcher::HTML, :clear, :convert_to_int
 
-      attr_reader :proxy
-
+      # Loads proxy provider page content, extract proxy list from it
+      # and convert every entry to proxy object.
       def fetch_proxies!(filters = {})
         load_proxy_list(filters).map { |html| to_proxy(html) }
       end
@@ -23,8 +23,10 @@ module ProxyFetcher
 
       # Loads HTML document with Nokogiri by the URL combined with custom filters
       def load_document(url, filters = {})
+        raise ArgumentError, 'filters must be a Hash' if filters && !filters.is_a?(Hash)
+
         uri = URI.parse(url)
-        uri.query = URI.encode_www_form(filters) if filters.any?
+        uri.query = URI.encode_www_form(filters) if filters && filters.any?
 
         Nokogiri::HTML(ProxyFetcher.config.http_client.fetch(uri.to_s))
       end

data/lib/proxy_fetcher/proxy.rb

@@ -11,12 +11,18 @@ module ProxyFetcher
 
     TYPES.each do |proxy_type|
       define_method "#{proxy_type.downcase}?" do
-        type && type.upcase.include?(proxy_type)
+        !type.nil? && type.upcase.include?(proxy_type)
       end
     end
 
     alias ssl? https?
 
+    def initialize(attributes = {})
+      attributes.each do |attr, value|
+        public_send("#{attr}=", value)
+      end
+    end
+
     def connectable?
       ProxyFetcher.config.proxy_validator.connectable?(addr, port)
     end

data/lib/proxy_fetcher/version.rb

@@ -7,9 +7,9 @@ module ProxyFetcher
     # Major version number
     MAJOR = 0
     # Minor version number
-    MINOR = 3
+    MINOR = 4
     # Smallest version number
-    TINY  = 1
+    TINY  = 0
 
     # Full version number
     STRING = [MAJOR, MINOR, TINY].compact.join('.')

data/proxy_fetcher.gemspec

@@ -5,7 +5,7 @@ require 'proxy_fetcher/version'
 Gem::Specification.new do |gem|
   gem.name = 'proxy_fetcher'
   gem.version = ProxyFetcher.gem_version
-  gem.date = '2017-08-21'
+  gem.date = '2017-08-28'
   gem.summary = 'Ruby gem for dealing with proxy lists from different providers'
   gem.description = 'This gem can help your Ruby application to make HTTP(S) requests ' \
                     'from proxy server by fetching and validating proxy lists from the different providers.'

data/spec/proxy_fetcher/configuration_spec.rb

@@ -43,21 +43,16 @@ describe ProxyFetcher::Configuration do
   end
 
   context 'custom provider' do
-    it 'successfully setups if provider class registered' do
-      CustomProvider = Class.new(ProxyFetcher::Providers::Base)
-      ProxyFetcher::Configuration.register_provider(:custom_provider, CustomProvider)
-
-      expect { ProxyFetcher.config.provider = :custom_provider }.not_to raise_error
+    it 'failed on registration if provider class already registered' do
+      expect { ProxyFetcher::Configuration.register_provider(:xroxy, Class.new) }
+        .to raise_error(ProxyFetcher::ProvidersRegistry::RegisteredProvider)
     end
 
-    it 'failed on setup if provider class is not registered' do
-      expect { ProxyFetcher.config.provider = :unexisting_provider }
-        .to raise_error(ProxyFetcher::Configuration::UnknownProvider)
-    end
+    it "failed on proxy list fetching if provider doesn't registered" do
+      ProxyFetcher.config.provider = :not_existing_provider
 
-    it 'failed on setup if provider class already registered' do
-      expect { ProxyFetcher::Configuration.register_provider(:xroxy, Class.new)}
-        .to raise_error(ProxyFetcher::Configuration::RegisteredProvider)
+      expect { ProxyFetcher::Manager.new }
+        .to raise_error(ProxyFetcher::ProvidersRegistry::UnknownProvider)
     end
   end
 end

data/spec/proxy_fetcher/providers/multiple_providers_spec.rb

@@ -0,0 +1,21 @@
+require 'spec_helper'
+
+describe 'Multiple proxy providers' do
+  before { ProxyFetcher.config.reset! }
+  after { ProxyFetcher.config.reset! }
+
+  it 'combine proxies from multiple providers' do
+    proxy_stub = ProxyFetcher::Proxy.new(addr: '192.168.1.1', port: 8080)
+
+    # Each proxy provider will return 2 proxies
+    ProxyFetcher::Configuration.providers_registry.providers.each do |_name, klass|
+      allow_any_instance_of(klass).to receive(:load_proxy_list).and_return([1, 2])
+      allow_any_instance_of(klass).to receive(:to_proxy).and_return(proxy_stub)
+    end
+
+    all_providers = ProxyFetcher::Configuration.registered_providers
+    ProxyFetcher.config.providers = all_providers
+
+    expect(ProxyFetcher::Manager.new.proxies.size).to eq(all_providers.size * 2)
+  end
+end

data/spec/proxy_fetcher/proxy_spec.rb

@@ -11,6 +11,15 @@ describe ProxyFetcher::Proxy do
 
   let(:proxy) { @manager.proxies.first.dup }
 
+  it 'can initialize a new proxy object' do
+    proxy = described_class.new(addr: '192.169.1.1', port: 8080, type: 'HTTP')
+
+    expect(proxy).not_to be_nil
+    expect(proxy.addr).to eq('192.169.1.1')
+    expect(proxy.port).to eq(8080)
+    expect(proxy.type).to eq('HTTP')
+  end
+
   it 'checks schema' do
     proxy.type = ProxyFetcher::Proxy::HTTP
     expect(proxy.http?).to be_truthy

data/spec/support/manager_examples.rb

@@ -9,9 +9,18 @@ RSpec.shared_examples 'a manager' do
     expect(manager.proxies).to be_empty
   end
 
-  it 'returns Proxy objects' do
+  it 'returns valid Proxy objects' do
     manager = ProxyFetcher::Manager.new
     expect(manager.proxies).to all(be_a(ProxyFetcher::Proxy))
+
+    manager.proxies.each do |proxy|
+      expect(proxy.addr).to match(/\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b/i)
+      expect(proxy.port).to be_a_kind_of(Numeric)
+      expect(proxy.type).not_to be_empty
+      expect(proxy.country).not_to be_empty
+      expect(proxy.anonymity).not_to be_empty
+      expect(proxy.response_time).to be_nil.or(be_a_kind_of(Numeric))
+    end
   end
 
   it 'returns raw proxies (HOST:PORT)' do

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: proxy_fetcher
 version: !ruby/object:Gem::Version
-  version: 0.3.1
+  version: 0.4.0
 platform: ruby
 authors:
 - Nikita Bulai
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-08-21 00:00:00.000000000 Z
+date: 2017-08-28 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: nokogiri
@@ -55,6 +55,8 @@ files:
 - ".gitignore"
 - ".rubocop.yml"
 - ".travis.yml"
+- CHANGELOG.md
+- CODE_OF_CONDUCT.md
 - Gemfile
 - LICENSE
 - README.md
@@ -62,6 +64,7 @@ files:
 - bin/proxy_fetcher
 - lib/proxy_fetcher.rb
 - lib/proxy_fetcher/configuration.rb
+- lib/proxy_fetcher/configuration/providers_registry.rb
 - lib/proxy_fetcher/manager.rb
 - lib/proxy_fetcher/providers/base.rb
 - lib/proxy_fetcher/providers/free_proxy_list.rb
@@ -81,6 +84,7 @@ files:
 - spec/proxy_fetcher/providers/free_proxy_list_spec.rb
 - spec/proxy_fetcher/providers/free_proxy_list_ssl_spec.rb
 - spec/proxy_fetcher/providers/hide_my_name_spec.rb
+- spec/proxy_fetcher/providers/multiple_providers_spec.rb
 - spec/proxy_fetcher/providers/proxy_docker_spec.rb
 - spec/proxy_fetcher/providers/proxy_list_spec.rb
 - spec/proxy_fetcher/providers/xroxy_spec.rb