proxy_fetcher 0.5.1 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: eb17af3b54c59182794340dd9e07351e3107c380
-  data.tar.gz: f86f209533d12f4a536ac4a5ae07d48a8afc9e90
+  metadata.gz: 6a8a3fe3140e235a3b46ecdb64322c77c8ce69d4
+  data.tar.gz: 9ace00e654e55832242e050ee42d01642b26338c
 SHA512:
-  metadata.gz: 0ecf1b6ee299550228ea651a220bcc9d85f3e383844974416862d567634a35d41d3650499ee6a8695af9b26c53fea3bba35a57336b1cfc795996df10c301751b
-  data.tar.gz: dcb39302fa9b2b43869be96484794d273a579546f92c27d5103f535d2fe8768220ab3a781d8373a58e11ebe5a2abe9efb13c772c32b088aa393bf890ca66c385
+  metadata.gz: 961dd103ae502f947a7417b248ba7cbfe8e5907880bbe2b2b123b568de6f703f39ba6a2da281dbb139a62703d9f2a26f4bb02be6128df6b09457542aa7235ba3
+  data.tar.gz: d80b53cdfb9f67c76edf4176ab6bd3daa8151d814f67e82405b53495ecab4be0d963232e010efe333a5ef1dfd177251367b9d620d9cc78306fccfc2ef137b837

data/.gitignore

@@ -15,6 +15,7 @@ pickle-email-*.html
 Gemfile.lock
 *.gem
 certs
+gemfiles/*.gemfile.lock
 
 # TODO Comment out this rule if you are OK with secrets being uploaded to the repo
 config/initializers/secret_token.rb

data/.rubocop.yml

@@ -1,7 +1,7 @@
 LineLength:
   Max: 120
 AllCops:
-  TargetRubyVersion: 2.4
+  TargetRubyVersion: 2.1
   Exclude:
     - 'spec/**/*'
     - 'bin/*'

data/.travis.yml

@@ -2,6 +2,12 @@ language: ruby
 before_install: gem install bundler
 bundler_args: --without yard guard benchmarks
 script: "rake spec"
+env:
+  global:
+    - "JRUBY_OPTS='$JRUBY_OPTS --debug'"
+gemfile:
+  - gemfiles/oga.gemfile
+  - gemfiles/nokogiri.gemfile
 rvm:
   - 2.0
   - 2.1
@@ -12,3 +18,6 @@ rvm:
 matrix:
   allow_failures:
     - rvm: ruby-head
+  exclude:
+    - rvm: 2.0
+      gemfile: gemfiles/nokogiri.gemfile # Nokogiri doesn't support Ruby 2.0

data/Gemfile

@@ -2,7 +2,10 @@ source 'https://rubygems.org'
 
 gemspec
 
+gem 'nokogiri', '~> 1.8'
+gem 'oga', '~> 2.0'
+
 group :test do
   gem 'coveralls', require: false
-  gem 'evil-proxy'
+  gem 'evil-proxy', '~> 0.2'
 end

data/README.md

@@ -5,17 +5,19 @@
 [![Code Climate](https://codeclimate.com/github/nbulaj/proxy_fetcher/badges/gpa.svg)](https://codeclimate.com/github/nbulaj/proxy_fetcher)
 [![License](http://img.shields.io/badge/license-MIT-brightgreen.svg)](#license)
 
-This gem can help your Ruby application to make HTTP(S) requests from proxy by fetching and validating actual
+This gem can help your Ruby application to make HTTP(S) requests using proxy by fetching and validating actual
 proxy lists from multiple providers.
 
-It gives you a `Manager` class that can load proxy lists, validate them and return random or specific proxies. Take a look
-at the documentation below to find all the gem features.
+It gives you a special `Manager` class that can load proxy lists, validate them and return random or specific proxies.
+It also has a `Client` class that encapsulates all the logic for the sending HTTP requests using proxies.
+Take a look at the documentation below to find all the gem features.
 
 Also this gem can be used with any other programming language (Go / Python / etc) as standalone solution for downloading and
 validating proxy lists from the different providers. [Checkout examples](#standalone) of usage below.
 
 ## Table of Contents
 
+- [Dependencies](#dependencies)
 - [Installation](#installation)
 - [Example of usage](#example-of-usage)
   - [In Ruby application](#in-ruby-application)
@@ -28,12 +30,24 @@ validating proxy lists from the different providers. [Checkout examples](#standa
 - [Contributing](#contributing)
 - [License](#license)
 
+## Dependencies
+
+ProxyFetcher gem itself requires only Ruby `>= 2.0.0`.
+
+However, it requires an adapter to parse HTML. If you do not specify any specific adapter, then it will use
+default one - [Nokogiri](https://github.com/sparklemotion/nokogiri). It's OK for any Ruby on Rails project
+(because they uses it by default).
+
+But if you want to use some specific adapter (for example your Ruby application uses [Oga](https://gitlab.com/yorickpeterse/oga),
+then you need to manually add your dependencies to your project and configure ProxyFetcher to use another adapter. Moreover,
+you can implement your own adapter if it your use-case. Take a look at the [Configuration](#configuration) section for more details.
+
 ## Installation
 
 If using bundler, first add 'proxy_fetcher' to your Gemfile:
 
 ```ruby
-gem 'proxy_fetcher', '~> 0.5'
+gem 'proxy_fetcher', '~> 0.6'
 ```
 
 or if you want to use the latest version (from `master` branch), then:
@@ -234,7 +248,25 @@ Btw, if you need support of JavaScript or some other features, you need to imple
 
 ## Configuration
 
-To change open/read timeout for `cleanup!` and `connectable?` methods you need to change `ProxyFetcher.config`:
+ProxyFetcher is very flexible gem. You can configure the most important parts of the library and use your own solutions.
+
+Default configuration looks as follows:
+
+```ruby
+ProxyFetcher.configure do |config|
+  config.user_agent = ProxyFetcher::Configuration::DEFAULT_USER_AGENT
+  config.pool_size = 10
+  config.timeout = 3
+  config.http_client = ProxyFetcher::HTTPClient
+  config.proxy_validator = ProxyFetcher::ProxyValidator
+  config.providers = ProxyFetcher::Configuration.registered_providers
+  config.adapter = ProxyFetcher::Configuration::DEFAULT_ADAPTER # :nokogiri by default
+end
+```
+
+You can change any of the options above. Let's look at this deeper.
+
+To change open/read timeout for `cleanup!` and `connectable?` methods you need to change `timeout` options:
 
 ```ruby
 ProxyFetcher.configure do |config|
@@ -245,7 +277,7 @@ manager = ProxyFetcher::Manager.new
 manager.cleanup!
 ```
 
-Also you can set your custom User-Agent:
+Also you can set your custom User-Agent string:
 
 ```ruby
 ProxyFetcher.configure do |config|
@@ -253,10 +285,11 @@ ProxyFetcher.configure do |config|
 end
 ```
 
-ProxyFetcher uses simple Ruby solution for dealing with HTTP(S) requests - `net/http` library from the stdlib. If you wanna add, for example, your custom provider that
-was developed as a Single Page Application (SPA) with some JavaScript, then you will need something like [selenium-webdriver](https://github.com/SeleniumHQ/selenium/tree/master/rb)
-to properly load the content of the website. For those and other cases you can write your own class for fetching HTML content by the URL and setup it
-in the ProxyFetcher config:
+ProxyFetcher uses standard Ruby solution for dealing with HTTP(S) requests - `net/http` library from the Ruby core.
+If you wanna add, for example, your custom provider that was developed as a Single Page Application (SPA) with some JavaScript,
+then you will need something like [selenium-webdriver](https://github.com/SeleniumHQ/selenium/tree/master/rb) to properly
+load the content of the website. For those and other cases you can write your own class for fetching HTML content by
+the URL and setup it in the ProxyFetcher config:
 
 ```ruby
 class MyHTTPClient
@@ -300,6 +333,21 @@ manager.validate!
  #=> [ ... ]
 ```
 
+Be default, ProxyFetcher gem uses [Nokogiri](https://github.com/sparklemotion/nokogiri) for parsing HTML. If you want
+to use [Oga](https://gitlab.com/yorickpeterse/oga) instead, then you need to add `gem 'oga'` to your Gemfile and configure
+ProxyFetcher as follows:
+
+```ruby
+ProxyFetcher.config.adapter = :oga
+```
+
+Also you can write your own HTML parser implementation and use it, take a look at the [abstract class and implementations](lib/proxy_fetcher/document).
+Configure it as:
+
+```ruby
+ProxyFetcher.config.adapter = MyHTMLParserClass
+```
+
 ### Proxy validation speed
 
 There are some tricks to increase proxy list validation performance.

data/gemfiles/nokogiri.gemfile

@@ -0,0 +1,11 @@
+source 'https://rubygems.org'
+
+gemspec path: '../'
+
+gem 'nokogiri', '~> 1.8'
+
+group :test do
+  gem 'coveralls', require: false
+  gem 'evil-proxy', '~> 0.2'
+  gem 'rspec-rails', '~> 3.6'
+end

data/gemfiles/oga.gemfile

@@ -0,0 +1,11 @@
+source 'https://rubygems.org'
+
+gemspec path: '../'
+
+gem 'oga', '~> 2.0'
+
+group :test do
+  gem 'coveralls', require: false
+  gem 'evil-proxy', '~> 0.2'
+  gem 'rspec-rails', '~> 3.6'
+end

data/lib/proxy_fetcher/configuration.rb

@@ -1,11 +1,13 @@
 module ProxyFetcher
   class Configuration
-    attr_accessor :providers, :timeout, :pool_size, :user_agent
-    attr_accessor :http_client, :proxy_validator
+    attr_accessor :timeout, :pool_size, :user_agent
+    attr_reader :adapter, :http_client, :proxy_validator, :providers
 
     # rubocop:disable Metrics/LineLength
     DEFAULT_USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112 Safari/537.36'.freeze
 
+    DEFAULT_ADAPTER = :nokogiri
+
     class << self
       def providers_registry
         @registry ||= ProvidersRegistry.new
@@ -35,6 +37,11 @@ module ProxyFetcher
       self.providers = self.class.registered_providers
     end
 
+    def adapter=(name_or_class)
+      @adapter = ProxyFetcher::Document::Adapters.lookup(name_or_class)
+      @adapter.setup!
+    end
+
     def providers=(value)
       @providers = Array(value)
     end

data/lib/proxy_fetcher/document/adapters/abstract_adapter.rb

@@ -0,0 +1,31 @@
+module ProxyFetcher
+  class Document
+    class AbstractAdapter
+      attr_reader :document
+
+      def initialize(document)
+        @document = document
+      end
+
+      # You can override this method in your own adapter class
+      def xpath(selector)
+        document.xpath(selector)
+      end
+
+      # You can override this method in your own adapter class
+      def css(selector)
+        document.css(selector)
+      end
+
+      def proxy_node
+        self.class.const_get('Node')
+      end
+
+      def self.setup!(*args)
+        install_requirements!(*args)
+      rescue LoadError => error
+        raise Exceptions::AdapterSetupError.new(name, error.message)
+      end
+    end
+  end
+end

data/lib/proxy_fetcher/document/adapters/nokogiri_adapter.rb

@@ -0,0 +1,35 @@
+module ProxyFetcher
+  class Document
+    class NokogiriAdapter < AbstractAdapter
+      def self.install_requirements!
+        require 'nokogiri'
+      end
+
+      def self.parse(data)
+        new(::Nokogiri::HTML(data))
+      end
+
+      class Node < ProxyFetcher::Document::Node
+        def at_xpath(*args)
+          self.class.new(node.at_xpath(*args))
+        end
+
+        def at_css(*args)
+          self.class.new(node.at_css(*args))
+        end
+
+        def attr(*args)
+          clear(node.attr(*args))
+        end
+
+        def content
+          clear(node.content)
+        end
+
+        def html
+          node.inner_html
+        end
+      end
+    end
+  end
+end

data/lib/proxy_fetcher/document/adapters/oga_adapter.rb

@@ -0,0 +1,35 @@
+module ProxyFetcher
+  class Document
+    class OgaAdapter < AbstractAdapter
+      def self.install_requirements!
+        require 'oga'
+      end
+
+      def self.parse(data)
+        new(::Oga.parse_html(data))
+      end
+
+      class Node < ProxyFetcher::Document::Node
+        def at_xpath(*args)
+          self.class.new(node.at_xpath(*args))
+        end
+
+        def at_css(*args)
+          self.class.new(node.at_css(*args))
+        end
+
+        def attr(*args)
+          clear(node.attribute(*args).value)
+        end
+
+        def content
+          clear(node.text)
+        end
+
+        def html
+          node.to_xml
+        end
+      end
+    end
+  end
+end

data/lib/proxy_fetcher/document/adapters.rb

@@ -0,0 +1,24 @@
+module ProxyFetcher
+  class Document
+    class Adapters
+      ADAPTER = 'Adapter'.freeze
+      private_constant :ADAPTER
+
+      class << self
+        def lookup(name_or_class)
+          raise Exceptions::BlankAdapter if name_or_class.nil? || name_or_class.to_s.empty?
+
+          case name_or_class
+          when Symbol, String
+            adapter_name = name_or_class.to_s.capitalize << ADAPTER
+            ProxyFetcher::Document.const_get(adapter_name)
+          else
+            name_or_class
+          end
+        rescue NameError
+          raise Exceptions::UnknownAdapter, name_or_class
+        end
+      end
+    end
+  end
+end

data/lib/proxy_fetcher/document/node.rb

@@ -0,0 +1,35 @@
+module ProxyFetcher
+  class Document
+    class Node
+      attr_reader :node
+
+      def initialize(node)
+        @node = node
+      end
+
+      def find(selector, method = :at_xpath)
+        self.class.new(node.public_send(method, selector))
+      end
+
+      def content_at(*args)
+        clear(find(*args).content)
+      end
+
+      def content
+        raise "#{__method__} must be implemented in descendant class!"
+      end
+
+      def html
+        raise "#{__method__} must be implemented in descendant class!"
+      end
+
+      protected
+
+      def clear(text)
+        return if text.nil? || text.empty?
+
+        text.strip.gsub(/[ \t]/i, '')
+      end
+    end
+  end
+end

data/lib/proxy_fetcher/document.rb

@@ -0,0 +1,23 @@
+module ProxyFetcher
+  class Document
+    class << self
+      def parse(data)
+        new(ProxyFetcher.config.adapter.parse(data))
+      end
+    end
+
+    attr_reader :backend
+
+    def initialize(backend)
+      @backend = backend
+    end
+
+    def xpath(*args)
+      backend.xpath(*args).map { |node| backend.proxy_node.new(node) }
+    end
+
+    def css(*args)
+      backend.css(*args).map { |node| backend.proxy_node.new(node) }
+    end
+  end
+end

data/lib/proxy_fetcher/exceptions.rb

@@ -32,5 +32,38 @@ module ProxyFetcher
         super('reached the maximum number of retries')
       end
     end
+
+    class UnknownAdapter < Error
+      def initialize(name)
+        super("unknown adapter '#{name}'")
+      end
+    end
+
+    class BlankAdapter < Error
+      def initialize(*)
+        super(<<-MSG.strip.squeeze
+          you need to specify adapter for HTML parsing: ProxyFetcher.config.adapter = :nokogiri.
+          You can use one of the predefined adapters (:nokogiri or :oga) or your own implementation.
+          MSG
+        )
+      end
+    end
+
+    class AdapterSetupError < Error
+      def initialize(adapter_name, reason)
+        adapter = demodulize(adapter_name.gsub('Adapter', ''))
+
+        super("can't setup '#{adapter}' adapter during the following error:\n\t#{reason}'")
+      end
+
+      private
+
+      def demodulize(path)
+        path = path.to_s
+        index = path.rindex('::')
+
+        index ? path[(index + 2)..-1] : path
+      end
+    end
   end
 end

data/lib/proxy_fetcher/providers/base.rb

@@ -1,12 +1,6 @@
-require 'forwardable'
-
 module ProxyFetcher
   module Providers
     class Base
-      extend Forwardable
-
-      def_delegators ProxyFetcher::HTML, :clear, :convert_to_int
-
       # Loads proxy provider page content, extract proxy list from it
       # and convert every entry to proxy object.
       def fetch_proxies!(filters = {})
@@ -14,8 +8,8 @@ module ProxyFetcher
       end
 
       class << self
-        def fetch_proxies!(filters = {})
-          new.fetch_proxies!(filters)
+        def fetch_proxies!(*args)
+          new.fetch_proxies!(*args)
         end
       end
 
@@ -23,12 +17,13 @@ module ProxyFetcher
 
       # Loads HTML document with Nokogiri by the URL combined with custom filters
       def load_document(url, filters = {})
-        raise ArgumentError, 'filters must be a Hash' if filters && !filters.is_a?(Hash)
+        raise ArgumentError, 'filters must be a Hash' unless filters.is_a?(Hash)
 
         uri = URI.parse(url)
         uri.query = URI.encode_www_form(filters) if filters && filters.any?
 
-        Nokogiri::HTML(ProxyFetcher.config.http_client.fetch(uri.to_s))
+        html = ProxyFetcher.config.http_client.fetch(uri.to_s)
+        ProxyFetcher::Document.parse(html)
       end
 
       # Get HTML elements with proxy info
@@ -40,11 +35,6 @@ module ProxyFetcher
       def to_proxy(*)
         raise NotImplementedError, "#{__method__} must be implemented in a descendant class!"
       end
-
-      # Return normalized HTML element content by selector
-      def parse_element(parent, selector, method = :at_xpath)
-        clear(parent.public_send(method, selector).content)
-      end
     end
   end
 end

data/lib/proxy_fetcher/providers/free_proxy_list.rb

@@ -9,20 +9,20 @@ module ProxyFetcher
         doc.xpath('//table[@id="proxylisttable"]/tbody/tr')
       end
 
-      def to_proxy(html_element)
+      def to_proxy(html_node)
         ProxyFetcher::Proxy.new.tap do |proxy|
-          proxy.addr = parse_element(html_element, 'td[1]')
-          proxy.port = convert_to_int(parse_element(html_element, 'td[2]'))
-          proxy.country = parse_element(html_element, 'td[4]')
-          proxy.anonymity = parse_element(html_element, 'td[5]')
-          proxy.type = parse_type(html_element)
+          proxy.addr = html_node.content_at('td[1]')
+          proxy.port = Integer(html_node.content_at('td[2]'))
+          proxy.country = html_node.content_at('td[4]')
+          proxy.anonymity = html_node.content_at('td[5]')
+          proxy.type = parse_type(html_node)
         end
       end
 
       private
 
-      def parse_type(element)
-        https = parse_element(element, 'td[6]')
+      def parse_type(html_node)
+        https = html_node.content_at('td[6]')
         https && https.casecmp('yes').zero? ? ProxyFetcher::Proxy::HTTPS : ProxyFetcher::Proxy::HTTP
       end
     end

data/lib/proxy_fetcher/providers/free_proxy_list_ssl.rb

@@ -9,12 +9,12 @@ module ProxyFetcher
         doc.xpath('//table[@id="proxylisttable"]/tbody/tr')
       end
 
-      def to_proxy(html_element)
+      def to_proxy(html_node)
         ProxyFetcher::Proxy.new.tap do |proxy|
-          proxy.addr = parse_element(html_element, 'td[1]')
-          proxy.port = convert_to_int(parse_element(html_element, 'td[2]'))
-          proxy.country = parse_element(html_element, 'td[4]')
-          proxy.anonymity = parse_element(html_element, 'td[5]')
+          proxy.addr = html_node.content_at('td[1]')
+          proxy.port = Integer(html_node.content_at('td[2]'))
+          proxy.country = html_node.content_at('td[4]')
+          proxy.anonymity = html_node.content_at('td[5]')
           proxy.type = ProxyFetcher::Proxy::HTTPS
         end
       end

data/lib/proxy_fetcher/providers/gather_proxy.rb

@@ -10,8 +10,8 @@ module ProxyFetcher
         doc.xpath('//div[@class="proxy-list"]/table/script')
       end
 
-      def to_proxy(html_element)
-        json = parse_json(html_element)
+      def to_proxy(html_node)
+        json = parse_json(html_node)
 
         ProxyFetcher::Proxy.new.tap do |proxy|
           proxy.addr = json['PROXY_IP']
@@ -25,8 +25,8 @@ module ProxyFetcher
 
       private
 
-      def parse_json(element)
-        javascript = clear(element.content)[/{.+}/im]
+      def parse_json(html_node)
+        javascript = html_node.content[/{.+}/im]
         JSON.parse(javascript)
       end
     end

data/lib/proxy_fetcher/providers/http_tunnel.rb

@@ -8,34 +8,34 @@ module ProxyFetcher
         doc.xpath('//table[contains(@id, "GridView")]/tr[(count(td)>2)]')
       end
 
-      def to_proxy(html_element)
+      def to_proxy(html_node)
         ProxyFetcher::Proxy.new.tap do |proxy|
-          uri = parse_proxy_uri(html_element)
+          uri = parse_proxy_uri(html_node)
           proxy.addr = uri.host
           proxy.port = uri.port
 
-          proxy.country = parse_country(html_element)
-          proxy.anonymity = parse_anonymity(html_element)
+          proxy.country = parse_country(html_node)
+          proxy.anonymity = parse_anonymity(html_node)
           proxy.type = ProxyFetcher::Proxy::HTTP
         end
       end
 
       private
 
-      def parse_proxy_uri(element)
-        full_addr = parse_element(element, 'td[1]')
+      def parse_proxy_uri(html_node)
+        full_addr = html_node.content_at('td[1]')
         URI.parse("http://#{full_addr}")
       end
 
-      def parse_country(element)
-        element.at('img').attr('title')
+      def parse_country(html_node)
+        html_node.find('.//img').attr('title')
       end
 
-      def parse_anonymity(element)
-        transparency = parse_element(element, 'td[5]').to_sym
+      def parse_anonymity(html_node)
+        transparency = html_node.content_at('td[5]').to_sym
 
         {
-          A: 'Anonimous',
+          A: 'Anonymous',
           E: 'Elite',
           T: 'Transparent',
           U: 'Unknown'

data/lib/proxy_fetcher/providers/proxy_docker.rb

@@ -9,15 +9,15 @@ module ProxyFetcher
         doc.xpath('//table[contains(@class, "table")]/tr[(not(@id="proxy-table-header")) and (count(td)>2)]')
       end
 
-      def to_proxy(html_element)
+      def to_proxy(html_node)
         ProxyFetcher::Proxy.new.tap do |proxy|
-          uri = URI("//#{parse_element(html_element, 'td[1]')}")
+          uri = URI("//#{html_node.content_at('td[1]')}")
           proxy.addr = uri.host
           proxy.port = uri.port
 
-          proxy.type = parse_element(html_element, 'td[2]')
-          proxy.anonymity = parse_element(html_element, 'td[3]')
-          proxy.country = parse_element(html_element, 'td[5]')
+          proxy.type = html_node.content_at('td[2]')
+          proxy.anonymity = html_node.content_at('td[3]')
+          proxy.country = html_node.content_at('td[5]')
         end
       end
     end

data/lib/proxy_fetcher/providers/proxy_list.rb

@@ -10,22 +10,22 @@ module ProxyFetcher
         doc.css('.table-wrap .table ul')
       end
 
-      def to_proxy(html_element)
+      def to_proxy(html_node)
         ProxyFetcher::Proxy.new.tap do |proxy|
-          uri = parse_proxy_uri(html_element)
+          uri = parse_proxy_uri(html_node)
           proxy.addr = uri.host
           proxy.port = uri.port
 
-          proxy.type = parse_element(html_element, 'li[2]')
-          proxy.anonymity = parse_element(html_element, 'li[4]')
-          proxy.country = clear(html_element.at_xpath("li[5]//span[@class='country']").attr('title'))
+          proxy.type = html_node.content_at('li[2]')
+          proxy.anonymity = html_node.content_at('li[4]')
+          proxy.country = html_node.find("li[5]//span[@class='country']").attr('title')
         end
       end
 
       private
 
-      def parse_proxy_uri(element)
-        full_addr = ::Base64.decode64(element.at('li script').inner_html.match(/'(.+)'/)[1])
+      def parse_proxy_uri(html_node)
+        full_addr = ::Base64.decode64(html_node.at_css('li script').html.match(/'(.+)'/)[1])
         URI.parse("http://#{full_addr}")
       end
     end

data/lib/proxy_fetcher/providers/xroxy.rb

@@ -8,21 +8,21 @@ module ProxyFetcher
         doc.xpath('//div[@id="content"]/table[1]/tr[contains(@class, "row")]')
       end
 
-      def to_proxy(html_element)
+      def to_proxy(html_node)
         ProxyFetcher::Proxy.new.tap do |proxy|
-          proxy.addr = parse_element(html_element, 'td[2]')
-          proxy.port = convert_to_int(parse_element(html_element, 'td[3]'))
-          proxy.anonymity = parse_element(html_element, 'td[4]')
-          proxy.country = parse_element(html_element, 'td[6]')
-          proxy.response_time = convert_to_int(parse_element(html_element, 'td[7]'))
-          proxy.type = parse_type(html_element)
+          proxy.addr = html_node.content_at('td[2]')
+          proxy.port = Integer(html_node.content_at('td[3]'))
+          proxy.anonymity = html_node.content_at('td[4]')
+          proxy.country = html_node.content_at('td[6]')
+          proxy.response_time = Integer(html_node.content_at('td[7]'))
+          proxy.type = parse_type(html_node)
         end
       end
 
       private
 
-      def parse_type(element)
-        https = parse_element(element, 'td[5]')
+      def parse_type(html_node)
+        https = html_node.content_at('td[5]')
         https.casecmp('true').zero? ? ProxyFetcher::Proxy::HTTPS : ProxyFetcher::Proxy::HTTP
       end
     end

data/lib/proxy_fetcher/version.rb

@@ -7,9 +7,9 @@ module ProxyFetcher
     # Major version number
     MAJOR = 0
     # Minor version number
-    MINOR = 5
+    MINOR = 6
     # Smallest version number
-    TINY  = 1
+    TINY  = 0
 
     # Full version number
     STRING = [MAJOR, MINOR, TINY].compact.join('.')

data/lib/proxy_fetcher.rb

@@ -1,7 +1,5 @@
 require 'uri'
 require 'net/https'
-require 'nokogiri'
-require 'thread'
 
 require File.dirname(__FILE__) + '/proxy_fetcher/exceptions'
 require File.dirname(__FILE__) + '/proxy_fetcher/configuration'
@@ -10,12 +8,18 @@ require File.dirname(__FILE__) + '/proxy_fetcher/proxy'
 require File.dirname(__FILE__) + '/proxy_fetcher/manager'
 
 require File.dirname(__FILE__) + '/proxy_fetcher/utils/http_client'
-require File.dirname(__FILE__) + '/proxy_fetcher/utils/html'
 require File.dirname(__FILE__) + '/proxy_fetcher/utils/proxy_validator'
 require File.dirname(__FILE__) + '/proxy_fetcher/client/client'
 require File.dirname(__FILE__) + '/proxy_fetcher/client/request'
 require File.dirname(__FILE__) + '/proxy_fetcher/client/proxies_registry'
 
+require File.dirname(__FILE__) + '/proxy_fetcher/document'
+require File.dirname(__FILE__) + '/proxy_fetcher/document/adapters'
+require File.dirname(__FILE__) + '/proxy_fetcher/document/node'
+require File.dirname(__FILE__) + '/proxy_fetcher/document/adapters/abstract_adapter'
+require File.dirname(__FILE__) + '/proxy_fetcher/document/adapters/nokogiri_adapter'
+require File.dirname(__FILE__) + '/proxy_fetcher/document/adapters/oga_adapter'
+
 module ProxyFetcher
   module Providers
     require File.dirname(__FILE__) + '/proxy_fetcher/providers/base'
@@ -36,5 +40,13 @@ module ProxyFetcher
     def configure
       yield config
     end
+
+    private
+
+    def configure_adapter!
+      config.adapter = Configuration::DEFAULT_ADAPTER if config.adapter.nil?
+    end
   end
+
+  configure_adapter!
 end

data/proxy_fetcher.gemspec

@@ -5,10 +5,10 @@ require 'proxy_fetcher/version'
 Gem::Specification.new do |gem|
   gem.name = 'proxy_fetcher'
   gem.version = ProxyFetcher.gem_version
-  gem.date = '2017-11-13'
+  gem.date = '2017-12-08'
   gem.summary = 'Ruby gem for dealing with proxy lists from different providers'
   gem.description = 'This gem can help your Ruby application to make HTTP(S) requests ' \
-                    'from proxy server by fetching and validating proxy lists from the different providers.'
+                    'using proxies by fetching and validating proxy lists from the different providers.'
   gem.authors = ['Nikita Bulai']
   gem.email = 'bulajnikita@gmail.com'
   gem.require_paths = ['lib']
@@ -19,7 +19,5 @@ Gem::Specification.new do |gem|
   gem.license = 'MIT'
   gem.required_ruby_version = '>= 2.0.0'
 
-  gem.add_runtime_dependency 'nokogiri', '~> 1.6', '>= 1.6'
-
   gem.add_development_dependency 'rspec', '~> 3.5'
 end

data/spec/proxy_fetcher/client_spec.rb

@@ -118,7 +118,7 @@ describe ProxyFetcher::Client do
     it 'refreshes proxy lists if no proxy found' do
       ProxyFetcher::Client::ProxiesRegistry.manager.instance_variable_set(:'@proxies', [])
 
-      expect { ProxyFetcher::Client.get('http://httpbin.org') }.not_to raise_error(ProxyFetcher::Exceptions::MaximumRetriesReached)
+      expect { ProxyFetcher::Client.get('http://httpbin.org') }.not_to raise_error
     end
   end
 

data/spec/proxy_fetcher/configuration_spec.rb

@@ -43,16 +43,33 @@ describe ProxyFetcher::Configuration do
   end
 
   context 'custom provider' do
-    it 'failed on registration if provider class already registered' do
+    it 'fails on registration if provider class already registered' do
       expect { ProxyFetcher::Configuration.register_provider(:xroxy, Class.new) }
         .to raise_error(ProxyFetcher::Exceptions::RegisteredProvider)
     end
 
-    it "failed on proxy list fetching if provider doesn't registered" do
+    it "fails on proxy list fetching if provider doesn't registered" do
       ProxyFetcher.config.provider = :not_existing_provider
 
       expect { ProxyFetcher::Manager.new }
         .to raise_error(ProxyFetcher::Exceptions::UnknownProvider)
     end
   end
+
+  context 'custom HTML parsing adapter' do
+    it "fails if adapter can't be installed" do
+      old_config = ProxyFetcher.config.dup
+
+      class CustomAdapter < ProxyFetcher::Document::AbstractAdapter
+        def self.install_requirements!
+          require 'not_existing_gem'
+        end
+      end
+
+      expect { ProxyFetcher.config.adapter = CustomAdapter }
+        .to raise_error(ProxyFetcher::Exceptions::AdapterSetupError)
+
+      ProxyFetcher.instance_variable_set('@config', old_config)
+    end
+  end
 end

data/spec/proxy_fetcher/document/adapters_spec.rb

@@ -0,0 +1,24 @@
+require 'spec_helper'
+
+describe ProxyFetcher::Document::Adapters do
+  describe '#lookup' do
+    it 'returns predefined adapters if symbol or string passed' do
+      expect(described_class.lookup('nokogiri')).to eq(ProxyFetcher::Document::NokogiriAdapter)
+
+      expect(described_class.lookup(:oga)).to eq(ProxyFetcher::Document::OgaAdapter)
+    end
+
+    it 'returns self if class passed' do
+      expect(described_class.lookup(Struct)).to eq(Struct)
+    end
+
+    it 'raises an exception if passed value is blank' do
+      expect { described_class.lookup(nil) }.to raise_error(ProxyFetcher::Exceptions::BlankAdapter)
+      expect { described_class.lookup('') }.to raise_error(ProxyFetcher::Exceptions::BlankAdapter)
+    end
+
+    it "raises an exception if adapter doesn't exist" do
+      expect { described_class.lookup('wrong') }.to raise_error(ProxyFetcher::Exceptions::UnknownAdapter)
+    end
+  end
+end

data/spec/spec_helper.rb

@@ -15,6 +15,13 @@ require 'proxy_fetcher'
 
 Dir['./spec/support/**/*.rb'].sort.each { |f| require f }
 
+adapter = ENV['BUNDLE_GEMFILE'][/.+\/(.+)\.gemfile/i, 1]
+puts "Configured adapter: '#{adapter}'"
+
+ProxyFetcher.configure do |config|
+  config.adapter = adapter
+end
+
 RSpec.configure do |config|
   config.order = 'random'
 end

metadata

@@ -1,35 +1,15 @@
 --- !ruby/object:Gem::Specification
 name: proxy_fetcher
 version: !ruby/object:Gem::Version
-  version: 0.5.1
+  version: 0.6.0
 platform: ruby
 authors:
 - Nikita Bulai
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-11-13 00:00:00.000000000 Z
+date: 2017-12-08 00:00:00.000000000 Z
 dependencies:
-- !ruby/object:Gem::Dependency
-  name: nokogiri
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.6'
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '1.6'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.6'
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '1.6'
 - !ruby/object:Gem::Dependency
   name: rspec
   requirement: !ruby/object:Gem::Requirement
@@ -44,8 +24,8 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '3.5'
-description: This gem can help your Ruby application to make HTTP(S) requests from
-  proxy server by fetching and validating proxy lists from the different providers.
+description: This gem can help your Ruby application to make HTTP(S) requests using
+  proxies by fetching and validating proxy lists from the different providers.
 email: bulajnikita@gmail.com
 executables:
 - proxy_fetcher
@@ -62,12 +42,20 @@ files:
 - README.md
 - Rakefile
 - bin/proxy_fetcher
+- gemfiles/nokogiri.gemfile
+- gemfiles/oga.gemfile
 - lib/proxy_fetcher.rb
 - lib/proxy_fetcher/client/client.rb
 - lib/proxy_fetcher/client/proxies_registry.rb
 - lib/proxy_fetcher/client/request.rb
 - lib/proxy_fetcher/configuration.rb
 - lib/proxy_fetcher/configuration/providers_registry.rb
+- lib/proxy_fetcher/document.rb
+- lib/proxy_fetcher/document/adapters.rb
+- lib/proxy_fetcher/document/adapters/abstract_adapter.rb
+- lib/proxy_fetcher/document/adapters/nokogiri_adapter.rb
+- lib/proxy_fetcher/document/adapters/oga_adapter.rb
+- lib/proxy_fetcher/document/node.rb
 - lib/proxy_fetcher/exceptions.rb
 - lib/proxy_fetcher/manager.rb
 - lib/proxy_fetcher/providers/base.rb
@@ -79,13 +67,13 @@ files:
 - lib/proxy_fetcher/providers/proxy_list.rb
 - lib/proxy_fetcher/providers/xroxy.rb
 - lib/proxy_fetcher/proxy.rb
-- lib/proxy_fetcher/utils/html.rb
 - lib/proxy_fetcher/utils/http_client.rb
 - lib/proxy_fetcher/utils/proxy_validator.rb
 - lib/proxy_fetcher/version.rb
 - proxy_fetcher.gemspec
 - spec/proxy_fetcher/client_spec.rb
 - spec/proxy_fetcher/configuration_spec.rb
+- spec/proxy_fetcher/document/adapters_spec.rb
 - spec/proxy_fetcher/providers/base_spec.rb
 - spec/proxy_fetcher/providers/free_proxy_list_spec.rb
 - spec/proxy_fetcher/providers/free_proxy_list_ssl_spec.rb
@@ -97,7 +85,6 @@ files:
 - spec/proxy_fetcher/providers/xroxy_spec.rb
 - spec/proxy_fetcher/proxy_spec.rb
 - spec/spec_helper.rb
-- spec/support/evil_proxy_patch.rb
 - spec/support/manager_examples.rb
 homepage: http://github.com/nbulaj/proxy_fetcher
 licenses:

data/lib/proxy_fetcher/utils/html.rb

@@ -1,15 +0,0 @@
-module ProxyFetcher
-  class HTML
-    class << self
-      def clear(text)
-        return if text.nil? || text.empty?
-
-        text.strip.gsub(/[ \t]/i, '')
-      end
-
-      def convert_to_int(text)
-        Integer(clear(text))
-      end
-    end
-  end
-end

data/spec/support/evil_proxy_patch.rb

@@ -1,26 +0,0 @@
-require 'evil-proxy'
-
-EvilProxy::HTTPProxyServer.class_eval do
-  def do_PUT(req, res)
-    perform_proxy_request(req, res) do |http, path, header|
-      http.put(path, req.body || '', header)
-    end
-  end
-
-  def do_DELETE(req, res)
-    perform_proxy_request(req, res) do |http, path, header|
-      http.delete(path, header)
-    end
-  end
-
-  def do_PATCH(req, res)
-    perform_proxy_request(req, res) do |http, path, header|
-      http.patch(path, req.body || '', header)
-    end
-  end
-
-# This method is not needed for PUT but I added for completeness
-  def do_OPTIONS(_req, res)
-    res['allow'] = 'GET,HEAD,POST,OPTIONS,CONNECT,PUT,PATCH,DELETE'
-  end
-end