proxy_fetcher 0.4.0 → 0.4.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: '0529f88d50000f7c2d3fad641372f5dcddfd40b3'
-  data.tar.gz: eab7966ac9aacf6cb62a8673280beed0fbb34330
+  metadata.gz: 40029164cd3a21183e8f22191e2cd8c01ba05ae3
+  data.tar.gz: ff557370ec0e68c0817fb009f9e483f936eaecd4
 SHA512:
-  metadata.gz: 45964603f17dc94e09075a1fb2264bc35a56a0667bcbc03a2142156414a23f35ee2d98aad1326fd64c80be7db8a31f5103244852253b56f50c168bb33fac94c1
-  data.tar.gz: 88ef9ef4e52e31277d0881f0b14b8ebf3095af18b993bee6cf1f0941edd56251a5ee3ea71f7b8ce2f725a127c8c6ef6f9d2c1b5c290103d43879b621cdd1813b
+  metadata.gz: 9a1461d4c42c3478675a4ad20e0d5e41a674d3486fe1026fdf4d95a1e484c4048f8f01476aaac14ccfa0c2a9e4e78ad2049e6b284828eeeb7c6670b1d0923630
+  data.tar.gz: 98f74fc50f15dc7f21f0f308494e4e874ad184cad512b401173da7ecc648c4fd0672aca45f2f3a4f0ec55aa6c01850d7979b579b8609a013d11c37a59443c10b

data/README.md

@@ -57,17 +57,20 @@ gem install proxy_fetcher -v '0.4'
 
 ### In Ruby application
 
-Get current proxy list without validation:
+By default ProxyFetcher uses all the available proxy providers. To get current proxy list without validation you
+need to initialize an instance of `ProxyFetcher::Manager` class. During this process ProxyFetcher will automatically load
+and parse all the proxies:
 
 ```ruby
 manager = ProxyFetcher::Manager.new # will immediately load proxy list from the server
 manager.proxies
 
- #=> [#<ProxyFetcher::Proxy:0x00000002879680 @addr="97.77.104.22", @port=3128, @country="USA", 
+ #=> [#<ProxyFetcher::Proxy:0x00000002879680 @addr="97.77.104.22", @port=3128, @country="USA",
  #     @response_time=5217, @type="HTTP", @anonymity="High">, ... ]
 ```
 
-You can initialize proxy manager without immediate load of proxy list from the remote server by passing `refresh: false` on initialization:
+You can initialize proxy manager without immediate load of the proxy list from the remote server by passing
+`refresh: false` on initialization:
 
 ```ruby
 manager = ProxyFetcher::Manager.new(refresh: false) # just initialize class instance
@@ -76,18 +79,27 @@ manager.proxies
  #=> []
 ```
 
-If you want to clean current proxy list from the dead servers that does not respond to the requests, than you can just call `cleanup!` method:
+`ProxyFetcher::Manager` class is very helpful when you need to manipulate and manager proxies. To get the proxy
+from the list yoy can call `.get` or `.pop` method that will return first proxy and move it to the end of the list.
+This methods has some equivalents like `get!` or aliased `pop!` that will return first **connectable** proxy and
+move it to the end of the list. They both marked as danger methods because all dead proxies will be removed from the list.
+
+If you need just some random proxy then call `manager.random_proxy` or it's alias `manager.random`.
+
+To clean current proxy list from the dead entries that does not respond to the requests you you need to use `cleanup!`
+or `validate!` method:
 
 ```ruby
 manager.cleanup! # or manager.validate!
 ```
 
-In order to increase the speed of this operation proxy list validation is performed using Ruby threads.
-By default, gem creates a pool with 10 threads, but you can increase this number by passing threads pool
-size to the `#cleanup!` (or `#validate!`) method: `manager.validate!(50)`. In that case ProxyFetcher will
-process all the fetched proxies in group of 50 threads.
+This action will enumerate proxy list and remove all the entries that doesn't respond by timeout or returns errors.
+
+In order to increase the performance proxy list validation is performed using Ruby threads. By default gem creates a
+pool with 10 threads, but you can increase this number by changing `pool_size` configuration option: `ProxyFetcher.config.pool_size = 50`.
+Read more in [Proxy validation speed](#proxy-validation-speed) section.
 
-Get raw proxy URLs as Strings:
+If you need raw proxy URLs (like `host:port`) then you can use `raw_proxies` methods that will return array of strings:
 
 ```ruby
 manager = ProxyFetcher::Manager.new
@@ -97,22 +109,16 @@ manager.raw_proxies
  #     "91.217.42.2:8080", "97.77.104.22:80", "165.234.102.177:8080", ...]
 ```
 
-If `ProxyFetcher::Manager` was already initialized somewhere, you can refresh the proxy list by calling `#refresh_list!` method:
+You don't need to initialize a new manager every time you want to load actual proxy list from the providers. All you
+need is to refresh the proxy list by calling `#refresh_list!` (or `#fetch!`) method for your `ProxyFetcher::Manager` instance:
 
 ```ruby
 manager.refresh_list! # or manager.fetch!
 
- #=> [#<ProxyFetcher::Proxy:0x00000002879680 @addr="97.77.104.22", @port=3128, @country="USA", 
+ #=> [#<ProxyFetcher::Proxy:0x00000002879680 @addr="97.77.104.22", @port=3128, @country="USA",
  #     @response_time=5217, @type="HTTP", @anonymity="High">, ... ]
 ```
 
-You can use two methods to get the first proxy from the list:
-
-* `get` or aliased `pop` (will return first proxy and move it to the end of the list)
-* `get!` or aliased `pop!` (will return first **connectable** proxy and move it to the end of the list; all the proxies till the working one will be removed)
-
-Or you can get just random proxy by calling `manager.random_proxy` or it's alias `manager.random`.
-
 If you need to filter proxy list, for example, by country or response time and selected provider supports filtering with GET params,
 then you can just pass your filters like a simple Ruby hash to the Manager instance:
 
@@ -139,7 +145,7 @@ manager = ProxyFetcher::Manager.new(filters: {
     type: 'All_http'
   }
 })
-  
+
 manager.proxies
 
  # => [...]
@@ -149,7 +155,7 @@ You can apply different filters every time you calling `#refresh_list!` (or `#fe
 
 ```ruby
 manager.refresh_list!(country: 'PL', maxtime: '500')
- 
+
  # => [...]
 ```
 
@@ -190,7 +196,7 @@ proxy_fetcher --help
 
 ## Configuration
 
-To change open/read timeout for `cleanup!` and `connectable?` methods you need to change ProxyFetcher.config:
+To change open/read timeout for `cleanup!` and `connectable?` methods you need to change `ProxyFetcher.config`:
 
 ```ruby
 ProxyFetcher.configure do |config|
@@ -219,7 +225,7 @@ ProxyFetcher.config.http_client = MyHTTPClient
 manager = ProxyFetcher::Manager.new
 manager.proxies
 
-#=> [#<ProxyFetcher::Proxy:0x00000002879680 @addr="97.77.104.22", @port=3128, @country="USA", 
+#=> [#<ProxyFetcher::Proxy:0x00000002879680 @addr="97.77.104.22", @port=3128, @country="USA",
  #     @response_time=5217, @type="HTTP", @anonymity="High">, ... ]
 ```
 
@@ -240,11 +246,11 @@ ProxyFetcher.config.proxy_validator = MyProxyValidator
 manager = ProxyFetcher::Manager.new
 manager.proxies
 
- #=> [#<ProxyFetcher::Proxy:0x00000002879680 @addr="97.77.104.22", @port=3128, @country="USA", 
+ #=> [#<ProxyFetcher::Proxy:0x00000002879680 @addr="97.77.104.22", @port=3128, @country="USA",
  #     @response_time=5217, @type="HTTP", @anonymity="High">, ... ]
- 
+
 manager.validate!
- 
+
  #=> [ ... ]
 ```
 
@@ -294,7 +300,7 @@ Also you can call next instance methods for every Proxy object:
 
 Currently ProxyFetcher can deal with next proxy providers (services):
 
-* Hide My Name (default one)
+* Hide My Name (**currently does not work**)
 * Free Proxy List
 * Free SSL Proxies
 * Proxy Docker

data/lib/proxy_fetcher/configuration.rb

@@ -30,7 +30,7 @@ module ProxyFetcher
       @http_client = HTTPClient
       @proxy_validator = ProxyValidator
 
-      self.providers = [:hide_my_name] # currently default one
+      self.providers = self.class.registered_providers
     end
 
     def providers=(value)

data/lib/proxy_fetcher/version.rb

@@ -9,7 +9,7 @@ module ProxyFetcher
     # Minor version number
     MINOR = 4
     # Smallest version number
-    TINY  = 0
+    TINY  = 1
 
     # Full version number
     STRING = [MAJOR, MINOR, TINY].compact.join('.')

data/proxy_fetcher.gemspec

@@ -5,7 +5,7 @@ require 'proxy_fetcher/version'
 Gem::Specification.new do |gem|
   gem.name = 'proxy_fetcher'
   gem.version = ProxyFetcher.gem_version
-  gem.date = '2017-08-28'
+  gem.date = '2017-09-04'
   gem.summary = 'Ruby gem for dealing with proxy lists from different providers'
   gem.description = 'This gem can help your Ruby application to make HTTP(S) requests ' \
                     'from proxy server by fetching and validating proxy lists from the different providers.'

data/spec/proxy_fetcher/providers/hide_my_name_spec.rb

@@ -5,5 +5,6 @@ describe ProxyFetcher::Providers::HideMyName do
     ProxyFetcher.config.provider = :hide_my_name
   end
 
-  it_behaves_like 'a manager'
+  # TODO: fix provider
+  # it_behaves_like 'a manager'
 end

data/spec/proxy_fetcher/proxy_spec.rb

@@ -2,7 +2,7 @@ require 'spec_helper'
 
 describe ProxyFetcher::Proxy do
   before :all do
-    ProxyFetcher.config.provider = :hide_my_name
+    ProxyFetcher.config.provider = :proxy_docker
   end
 
   before do

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: proxy_fetcher
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.4.1
 platform: ruby
 authors:
 - Nikita Bulai
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-08-28 00:00:00.000000000 Z
+date: 2017-09-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: nokogiri