vessel 0.1.0 → 0.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f52e40dc5f4086394068c24ed878e3b2bb82da411a33f1d4dc73c056bc84e53f
-  data.tar.gz: 17e3ee52630ecb8ac30f7ec75f36d0d92f160b9a817458db5ca00920684329f9
+  metadata.gz: 44fb472d4afaf916edc97894dcc39cf8b6bfbf3f8f1f0b2e8a47f495482b1bd9
+  data.tar.gz: 36af4cd9021bd410bf1988c01f97d98df4e5646f5a56416004869fe643403672
 SHA512:
-  metadata.gz: 1dff310db5f5daa97ece87535e6084a1e67b75f2c2a2476ee33aadb87895b52e32c991639826fef1188f1674cb86ffed32ff65df33d95cf9e5cd92a7baabd976
-  data.tar.gz: 4168d33ba27a09d4642cdd5551e7e6945d42f102aba3dc54ef622cc4281441c0b52b855a54dc0b4a9d1a07735def04b9d14ffd4fb1fe99c61746cf1803da7a08
+  metadata.gz: bda3863083cdce0e8011675a0e83a583d626e81ab713803a54c5056f922d4822b069dacd8d4e5f0079d4f8625a172f7f9d30d4e3586439137af088ac0911201e
+  data.tar.gz: 205b2f54fa17283daf50d0fdaa96e67f5dec4bed2c69ccc740433c90ecefaa9c4b1e13740cb703ac56cfe8c92b2df9da436fee94fc7937242465a33e91a088f5

data/README.md

@@ -1,17 +1,119 @@
 # Vessel - high-level web crawling framework
 
-## Installation
+#### Fast as Chrome, dead simple and yet extendable.
 
-Add this line to your application's Gemfile:
+It is Ruby high-level web crawling framework based on
+[Ferrum](https://github.com/rubycdp/ferrum) for extracting the data you need
+from websites. It can be used in a wide range of scenarios, like data mining,
+monitoring or historical archival. For automated testing we recommend
+[Cuprite](https://github.com/rubycdp/cuprite).
+
+Thanks to Evrone [design team](https://evrone.com/design?utm_source=github&utm_campaign=vessel). Read about [Vessel](https://evrone.com/vessel-framework?utm_source=github&utm_campaign=vessel) & other projects supported by Evrone [here](https://evrone.com/cases?utm_source=github&utm_campaign=vessel#open-source).
+
+
+## Install
+
+Add this to your Gemfile:
 
 ```ruby
 gem "vessel"
 ```
 
-And then execute:
 
-    $ bundle
+## A look around
+
+In order to show you how Vessel works we are going to crawl together
+[famous quotes website](http://quotes.toscrape.com):
+
+```ruby
+require "json"
+require "vessel"
+
+class QuotesToScrapeCom < Vessel::Cargo
+  domain "quotes.toscrape.com"
+  start_urls "http://quotes.toscrape.com/tag/humor/"
+
+  def parse
+    css("div.quote").each do |quote|
+      yield({
+        author: quote.at_xpath("span/small").text,
+        text: quote.at_css("span.text").text
+      })
+    end
+
+    if next_page = at_xpath("//li[@class='next']/a[@href]")
+      url = absolute_url(next_page.attribute(:href))
+      yield request(url: url, method: :parse)
+    end
+  end
+end
+
+quotes = []
+QuotesToScrapeCom.run { |q| quotes << q }
+puts JSON.generate(quotes)
+```
+
+Save this to `quotes.rb` file and run `bundle exec ruby quotes.rb > quotes.json`.
+When this finishes you will have a list of the quotes in JSON format in the
+`quotes.json` file.
+
+How it all works? First Vessel using Ferrum spawns Chrome which goes to one or
+more urls in `start_urls`, in our case it's only one. After Chrome reports back
+that page is loaded with all the resources it needs the first default callback
+`parse` is invoked. In the parse callback, we loop through the quote elements
+using a CSS Selector, yield a Hash with the extracted quote text and author and
+look for a link to the next page and schedule another request using the same
+parse method as callback.
+
+Notice that all requests are scheduled and handled concurrently. We use thread
+pool to work with all your requests with one page per core by default or add
+`threads max: n` to a class. If you yield more than one request Ruby will send
+them to Chrome which will load pages in parallel. Thus crawler is lightweight
+and speedy.
+
+
+## Settings
+
+* domain
+* start_urls
+* delay
+* timeout
+* threads
+* middleware
+
+
+## Selectors
+
+* at_css
+* css
+* at_xpath
+* xpath
+
+
+## Middleware
+
+To be continued
+
+
+## License
+
+Copyright 2018-2020 Machinio
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
 
-Or install it yourself as:
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
 
-    $ gem install vessel
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/lib/vessel.rb

@@ -1,6 +1,14 @@
+# frozen_string_literal: true
+
+require "concurrent-ruby"
+require "vessel/engine"
+require "vessel/middleware"
+require "vessel/scheduler"
+require "vessel/request"
 require "vessel/version"
+require "vessel/cargo"
 
 module Vessel
   class Error < StandardError; end
-  # Your code goes here...
+  class NotImplementedError < Error; end
 end

data/lib/vessel/cargo.rb

@@ -0,0 +1,86 @@
+# frozen_string_literal: true
+
+require "ferrum"
+require "forwardable"
+
+module Vessel
+  class Cargo
+    DELAY = 0
+    START_URLS = [].freeze
+    MIDDLEWARE = [].freeze
+    MIN_THREADS = 1
+    MAX_THREADS = Concurrent.processor_count
+
+    class << self
+      attr_reader :settings
+
+      def run(settings = nil, &block)
+        self.settings.merge!(Hash(settings))
+        Engine.run(self, &block)
+      end
+
+      def domain(name)
+        settings[:domain] = name
+      end
+
+      def start_urls(*urls)
+        settings[:start_urls] = urls
+      end
+
+      def delay(value)
+        settings[:delay] = value
+      end
+
+      def timeout(value)
+        settings[:timeout] = value
+      end
+
+      def threads(min: MIN_THREADS, max: MAX_THREADS)
+        settings[:min_threads] = min
+        settings[:max_threads] = max
+      end
+
+      def middleware(*classes)
+        settings[:middleware] = classes
+      end
+
+      def settings
+        @settings ||= {
+          delay: DELAY,
+          middleware: MIDDLEWARE,
+          start_urls: START_URLS,
+          min_threads: MIN_THREADS,
+          max_threads: MAX_THREADS,
+          domain: name&.split('::')&.last&.downcase
+        }
+      end
+    end
+
+    extend Forwardable
+    delegate %i[at_css css at_xpath xpath] => :page
+
+    attr_reader :page
+
+    def initialize(page = nil)
+      @page = page
+    end
+
+    def domain
+      self.class.settings[:domain]
+    end
+
+    def parse
+      raise NotImplementedError
+    end
+
+    private
+
+    def request(**options)
+      Request.new(**options)
+    end
+
+    def absolute_url(relative)
+      Addressable::URI.join(page.current_url, relative).to_s
+    end
+  end
+end

data/lib/vessel/cli.rb

@@ -0,0 +1,15 @@
+require "thor"
+
+module Vessel
+  class CLI < Thor
+    desc "version", "Print version."
+    def version
+      puts Vessel::VERSION
+    end
+
+    desc "start NAME", "Run given crawler."
+    def start(name)
+      raise NotImplementedError
+    end
+  end
+end

data/lib/vessel/engine.rb

@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+
+module Vessel
+  class Engine
+    def self.run(*args, &block)
+      new(*args, &block).tap(&:run)
+    end
+
+    attr_reader :crawler_class, :settings, :scheduler, :middleware
+
+    def initialize(klass, &block)
+      @crawler_class = klass
+      @settings = klass.settings
+      @middleware = block || Middleware.build(*settings[:middleware])
+      @queue = SizedQueue.new(settings[:max_threads])
+      @scheduler = Scheduler.new(@queue, settings)
+    end
+
+    def run
+      scheduler.post(*start_requests)
+
+      until @queue.closed?
+        message = @queue.pop
+        raise(message) if message.is_a?(Exception)
+        handle(*message)
+        @queue.close if idle?
+      end
+    end
+
+    def handle(page, request)
+      crawler = @crawler_class.new(page)
+      crawler.send(request.method) do |*args|
+        if args.all? { |i| i.is_a?(Request) }
+          scheduler.post(*args)
+        else
+          @middleware&.call(*args)
+        end
+      end
+    ensure
+      page.close
+    end
+
+    def start_requests
+      Request.build(*settings[:start_urls])
+    end
+
+    def idle?
+      @queue.empty? &&
+      @scheduler.queue_length.zero? &&
+      @scheduler.scheduled_task_count == @scheduler.completed_task_count
+    end
+  end
+end

data/lib/vessel/middleware.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+module Vessel
+  class Middleware
+    attr_reader :middleware
+
+    def self.build(*classes)
+      classes.inject { |base, klass| base.new(klass.new) }
+    end
+
+    def initialize(middleware = nil)
+      @middleware = middleware
+    end
+
+    def ==(other)
+      self.class == other.class
+    end
+
+    def call
+      raise NotImplementedError
+    end
+  end
+end

data/lib/vessel/request.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+require "addressable/uri"
+
+module Vessel
+  class Request
+    attr_reader :url, :uri, :method
+
+    def self.build(*urls)
+      urls.map { |url| new(url: url) }
+    end
+
+    def initialize(url:, method: :parse)
+      @url = url.to_s
+      @uri = Addressable::URI.parse(@url)
+      @method = method
+    end
+  end
+end

data/lib/vessel/scheduler.rb

@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+
+require "forwardable"
+require "concurrent-ruby"
+
+module Vessel
+  class Scheduler
+    extend Forwardable
+    delegate %i[scheduled_task_count completed_task_count queue_length] => :@pool
+
+    attr_reader :browser, :queue, :delay
+
+    def initialize(queue, settings)
+      @queue = queue
+      @min_threads, @max_threads, @delay =
+        settings.values_at(:min_threads, :max_threads, :delay)
+
+      options = {}
+      options.merge!(timeout: settings[:timeout]) if settings[:timeout]
+      @browser = Ferrum::Browser.new(**options)
+    end
+
+    def post(*requests)
+      requests.map do |request|
+        Concurrent::Promises.future_on(pool, queue, request) do |queue, request|
+          queue << goto(request)
+        end
+      end
+    end
+
+    private
+
+    def pool
+      @pool ||= Concurrent::ThreadPoolExecutor.new(
+        max_queue: 0,
+        min_threads: @min_threads,
+        max_threads: @max_threads
+      )
+    end
+
+    def goto(request)
+      page = browser.create_page
+      # Delay is set between requests when we don't want to bombard server with
+      # requests so it requires crawler to be single threaded. Otherwise doesn't
+      # make sense.
+      sleep(delay) if @max_threads == 1 && delay > 0
+      page.goto(request.url)
+      [page, request]
+    rescue => e
+      e
+    end
+  end
+end

data/lib/vessel/version.rb

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 module Vessel
-  VERSION = "0.1.0"
+  VERSION = "0.1.1"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: vessel
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.1.1
 platform: ruby
 authors:
 - Dmitry Vorotilin
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-09-17 00:00:00.000000000 Z
+date: 2020-04-06 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: ferrum
@@ -24,6 +24,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0.4'
+- !ruby/object:Gem::Dependency
+  name: thor
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.20'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.20'
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
@@ -77,6 +91,12 @@ files:
 - LICENSE
 - README.md
 - lib/vessel.rb
+- lib/vessel/cargo.rb
+- lib/vessel/cli.rb
+- lib/vessel/engine.rb
+- lib/vessel/middleware.rb
+- lib/vessel/request.rb
+- lib/vessel/scheduler.rb
 - lib/vessel/version.rb
 homepage: https://github.com/route/vessel
 licenses: