kimurai 1.1.0 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 67ee49692e64813bc980eb7562b711d5b5d2c47b50a995acb4759709703da0f9
-  data.tar.gz: baba361bc5039d303ae4a6c9a1dd2109368f8e4c7a641d0a782cfc6a7776ade4
+  metadata.gz: 2ebb62d0916ee55eb8ec05f34b5f5909b99aae8fa2236dd93169f6e5e1221805
+  data.tar.gz: da86497a6c4f61f2ff1ffe3e8eeee92acb203305e712b9bd4baaa5794fdcf5f5
 SHA512:
-  metadata.gz: 0173d3859b5f8776371fad454ff5575fdc453aa3c6038d8a8399651c46c5eaae789273772227ea014b6ce39b13586e6805bd7f69156eafeacf653804f954003c
-  data.tar.gz: b05889c0cb030aed06fe1df5cc5411154d24019667e1f00f9f4248d598fc93990f86a4aae78430af3140f3dc3989e856cfc3e2316f455984c898442fccad15db
+  metadata.gz: 9d54d6074928a5bc0aa0a9d7e64308942e21ad5ae17788b2fa36adfffe88f67df0c533558ced412f2e22481a1664661c547419d09451d1260dd6ebd14ca4d915
+  data.tar.gz: d21f499a2a292dd672480d15da71742cfa82dec054b6a3c3a3c756e6cd2c98e58ac7d3b7fa3fe0ce6ae2e956d46417269242e742f8d83b644176ccef0822de75

data/CHANGELOG.md

@@ -1,4 +1,20 @@
 # CHANGELOG
+## 1.2.0
+### New
+* Add possibility to add array of values to the storage (`Base::Storage#add`)
+* Add `exception_on_fail` option to `Base.crawl!`
+* Add possibility to pass request hash to the `start_urls` (You can use array of hashes as well, like: `@start_urls = [{ url: "https://example.com/cat?id=1", data: { category: "First Category" } }]`)
+* Implement `skip_request_errors` config feature. Added [Handle request errors](https://github.com/vifreefly/kimuraframework#handle-request-errors) chapter to the README.
+* Add option to choose response type for `Session#current_response` (`:html` default, or `:json`)
+* Add option to provide custom chrome and chromedriver paths
+
+### Improvements
+* Refactor `Runner`
+
+### Fixes
+* Fix `Base#Saver` (automatically create file if it doesn't exists in case of persistence database)
+* Do not deep merge config's `headers:` option
+
 ## 1.1.0
 ### Breaking changes 1.1.0
 `browser` config option depricated. Now all sub-options inside `browser` should be placed right into `@config` hash, without `browser` parent key. Example:

data/README.md

@@ -18,7 +18,7 @@
 
 <br>
 
-> Note: this readme is for `1.1.0` gem version. CHANGELOG [here](CHANGELOG.md).
+> Note: this readme is for `1.2.0` gem version. CHANGELOG [here](CHANGELOG.md).
 
 Kimurai is a modern web scraping framework written in Ruby which **works out of box with Headless Chromium/Firefox, PhantomJS**, or simple HTTP requests and **allows to scrape and interact with JavaScript rendered websites.**
 
@@ -216,7 +216,7 @@ I, [2018-08-22 13:33:30 +0400#23356] [M: 47375890851320]  INFO -- infinite_scrol
 * All the power of [Capybara](https://github.com/teamcapybara/capybara): use methods like `click_on`, `fill_in`, `select`, `choose`, `set`, `go_back`, etc. to interact with web pages
 * Rich [configuration](#spider-config): **set default headers, cookies, delay between requests, enable proxy/user-agents rotation**
 * Built-in helpers to make scraping easy, like [save_to](#save_to-helper) (save items to JSON, JSON lines, or CSV formats) or [unique?](#skip-duplicates-unique-helper) to skip duplicates
-* Automatically [retry failed requests](#configuration-options) with delay
+* Automatically [handle requests errors](#handle-request-errors)
 * Automatically restart browsers when reaching memory limit [**(memory control)**](#spider-config) or requests limit
 * Easily [schedule spiders](#schedule-spiders-using-cron) within cron using [Whenever](https://github.com/javan/whenever) (no need to know cron syntax)
 * [Parallel scraping](#parallel-crawling-using-in_parallel) using simple method `in_parallel`
@@ -242,6 +242,9 @@ I, [2018-08-22 13:33:30 +0400#23356] [M: 47375890851320]  INFO -- infinite_scrol
       * [Automatically skip all duplicated requests urls](#automatically-skip-all-duplicated-requests-urls)
       * [Storage object](#storage-object)
       * [Persistence database for the storage](#persistence-database-for-the-storage)
+    * [Handle request errors](#handle-request-errors)
+      * [skip_request_errors](#skip_request_errors)
+      * [retry_request_errors](#retry_request_errors)
     * [open_spider and close_spider callbacks](#open_spider-and-close_spider-callbacks)
     * [KIMURAI_ENV](#kimurai_env)
     * [Parallel crawling using in_parallel](#parallel-crawling-using-in_parallel)
@@ -860,6 +863,9 @@ ProductsSpider.crawl!(continue: true)
 
 Second approach is to automatically skip already processed items urls using `@config` `skip_duplicate_requests:` option:
 
+<details/>
+  <summary>Check the code</summary>
+
 ```ruby
 class ProductsSpider < Kimurai::Base
   @start_urls = ["https://example-shop.com/"]
@@ -893,7 +899,29 @@ end
 # Run the spider with persistence database option:
 ProductsSpider.crawl!(continue: true)
 ```
+</details>
+
+### Handle request errors
+It is quite common that some pages of crawling website can return different response code than `200 ok`. In such cases, method `request_to` (or `browser.visit`) can raise an exception. Kimurai provides `skip_request_errors` and `retry_request_errors` [config](#spider-config) options to handle such errors:
+
+#### skip_request_errors
+You can automatically skip some of errors while requesting a page using `skip_request_errors` [config](#spider-config) option. If raised error matches one of the errors in the list, then this error will be caught, and request will be skipped. It is a good idea to skip errors like NotFound(404), etc.
+
+Format for the option: array where elements are error classes or/and hashes. You can use _hash_ format for more flexibility:
+
+```ruby
+@config = {
+  skip_request_errors: [{ error: "RuntimeError", message: "404 => Net::HTTPNotFound" }]
+}
+```
+In this case, provided `message:` will be compared with a full error message using `String#include?`. Also you can use regex instead: `{ error: "RuntimeError", message: /404|403/ }`.
 
+#### retry_request_errors
+You can automatically retry some of errors with a few attempts while requesting a page using `retry_request_errors` [config](#spider-config) option. If raised error matches one of the errors in the list, then this error will be caught and the request will be processed again within a delay.
+
+There are 3 attempts: first: delay _15 sec_, second: delay _30 sec_, third: delay _45 sec_. If after 3 attempts there is still an exception, then the exception will be raised. It is a good idea to try to retry errros like `ReadTimeout`, `HTTPBadGateway`, etc.
+
+Format for the option: same like for `skip_request_errors` option.
 
 ### `open_spider` and `close_spider` callbacks
 
@@ -1340,6 +1368,11 @@ Kimurai.configure do |config|
   # Custom time zone (for logs):
   # config.time_zone = "UTC"
   # config.time_zone = "Europe/Moscow"
+
+  # Provide custom chrome binary path (default is any available chrome/chromium in the PATH):
+  # config.selenium_chrome_path = "/usr/bin/chromium-browser"
+  # Provide custom selenium chromedriver path (default is "/usr/local/bin/chromedriver"):
+  # config.chromedriver_path = "~/.local/bin/chromedriver"
 end
 ```
 
@@ -1591,7 +1624,23 @@ end
   # works for all drivers
   skip_duplicate_requests: true,
 
-  # Array of possible errors to retry while processing a request:
+  # Automatically skip provided errors while requesting a page.
+  # If raised error matches one of the errors in the list, then this error will be caught,
+  # and request will be skipped.
+  # It is a good idea to skip errors like NotFound(404), etc.
+  # Format: array where elements are error classes or/and hashes. You can use hash format
+  # for more flexibility: `{ error: "RuntimeError", message: "404 => Net::HTTPNotFound" }`.
+  # Provided `message:` will be compared with a full error message using `String#include?`. Also
+  # you can use regex instead: `{ error: "RuntimeError", message: /404|403/ }`.
+  skip_request_errors: [{ error: RuntimeError, message: "404 => Net::HTTPNotFound" }],
+
+  # Automatically retry provided errors with a few attempts while requesting a page.
+  # If raised error matches one of the errors in the list, then this error will be caught
+  # and the request will be processed again within a delay. There are 3 attempts:
+  # first: delay 15 sec, second: delay 30 sec, third: delay 45 sec.
+  # If after 3 attempts there is still an exception, then the exception will be raised.
+  # It is a good idea to try to retry errros like `ReadTimeout`, `HTTPBadGateway`, etc.
+  # Format: same like for `skip_request_errors` option.
   retry_request_errors: [Net::ReadTimeout],
 
   # Restart browser if one of the options is true:
@@ -1602,6 +1651,8 @@ end
     # Restart browser if provided requests limit is exceeded (works for all engines)
     requests_limit: 100
   },
+
+  # Perform several actions before each request:
   before_request: {
     # Change proxy before each request. The `proxy:` option above should be presented
     # and has lambda format. Works only for poltergeist and mechanize engines

data/lib/kimurai.rb

@@ -10,6 +10,7 @@ require_relative 'kimurai/version'
 require_relative 'kimurai/core_ext/numeric'
 require_relative 'kimurai/core_ext/string'
 require_relative 'kimurai/core_ext/array'
+require_relative 'kimurai/core_ext/hash'
 
 require_relative 'kimurai/browser_builder'
 require_relative 'kimurai/base_helper'

data/lib/kimurai/base.rb

@@ -3,6 +3,9 @@ require_relative 'base/storage'
 
 module Kimurai
   class Base
+    # don't deep merge config's headers hash option
+    DMERGE_EXCLUDE = [:headers]
+
     LoggerFormatter = proc do |severity, datetime, progname, msg|
       current_thread_id = Thread.current.object_id
       thread_type = Thread.main == Thread.current ? "M" : "C"
@@ -77,7 +80,11 @@ module Kimurai
     end
 
     def self.config
-      superclass.equal?(::Object) ? @config : superclass.config.deep_merge(@config || {})
+      if superclass.equal?(::Object)
+        @config
+      else
+        superclass.config.deep_merge_excl(@config || {}, DMERGE_EXCLUDE)
+      end
     end
 
     ###
@@ -90,7 +97,7 @@ module Kimurai
       end
     end
 
-    def self.crawl!(continue: false)
+    def self.crawl!(continue: false, exception_on_fail: true)
       logger.error "Spider: already running: #{name}" and return false if running?
 
       storage_path =
@@ -118,19 +125,23 @@ module Kimurai
       spider.with_info = true
       if start_urls
         start_urls.each do |start_url|
-          spider.request_to(:parse, url: start_url)
+          if start_url.class == Hash
+            spider.request_to(:parse, start_url)
+          else
+            spider.request_to(:parse, url: start_url)
+          end
         end
       else
         spider.parse
       end
     rescue StandardError, SignalException, SystemExit => e
       @run_info.merge!(status: :failed, error: e.inspect)
-      raise e
+      exception_on_fail ? raise(e) : [@run_info, e]
     else
       @run_info.merge!(status: :completed)
     ensure
       if spider
-        spider.browser.destroy_driver!
+        spider.browser.destroy_driver! if spider.instance_variable_get("@browser")
 
         stop_time  = Time.now
         total_time = (stop_time - @run_info[:start_time]).round(3)
@@ -168,7 +179,7 @@ module Kimurai
 
     def initialize(engine = self.class.engine, config: {})
       @engine = engine
-      @config = self.class.config.deep_merge(config)
+      @config = self.class.config.deep_merge_excl(config, DMERGE_EXCLUDE)
       @pipelines = self.class.pipelines.map do |pipeline_name|
         klass = Pipeline.descendants.find { |kl| kl.name == pipeline_name }
         instance = klass.new
@@ -184,15 +195,16 @@ module Kimurai
       @browser ||= BrowserBuilder.build(@engine, @config, spider: self)
     end
 
-    def request_to(handler, delay = nil, url:, data: {})
+    def request_to(handler, delay = nil, url:, data: {}, response_type: :html)
       if @config[:skip_duplicate_requests] && !unique_request?(url)
         add_event(:duplicate_requests) if self.with_info
-        logger.warn "Spider: request_to: url is not unique: #{url}, skipped" and return
+        logger.warn "Spider: request_to: not unique url: #{url}, skipped" and return
       end
 
-      request_data = { url: url, data: data }
-      delay ? browser.visit(url, delay: delay) : browser.visit(url)
-      public_send(handler, browser.current_response, request_data)
+      visited = delay ? browser.visit(url, delay: delay) : browser.visit(url)
+      return unless visited
+
+      public_send(handler, browser.current_response(response_type), { url: url, data: data })
     end
 
     def console(response = nil, url: nil, data: {})
@@ -285,18 +297,22 @@ module Kimurai
         all << Thread.new(part) do |part|
           Thread.current.abort_on_exception = true
 
-          spider = self.class.new(engine, config: config)
+          spider = self.class.new(engine, config: @config.deep_merge_excl(config, DMERGE_EXCLUDE))
           spider.with_info = true if self.with_info
 
           part.each do |url_data|
             if url_data.class == Hash
-              spider.request_to(handler, delay, url_data)
+              if url_data[:url].present? && url_data[:data].present?
+                spider.request_to(handler, delay, url_data)
+              else
+                spider.public_send(handler, url_data)
+              end
             else
               spider.request_to(handler, delay, url: url_data, data: data)
             end
           end
         ensure
-          spider.browser.destroy_driver!
+          spider.browser.destroy_driver! if spider.instance_variable_get("@browser")
         end
 
         sleep 0.5

data/lib/kimurai/base/saver.rb

@@ -42,7 +42,7 @@ module Kimurai
       def save_to_json(item)
         data = JSON.generate([item])
 
-        if append || @index > 1
+        if @index > 1 || append && File.exists?(path)
           file_content = File.read(path).sub(/\}\]\Z/, "\}\,")
           File.open(path, "w") do |f|
             f.write(file_content + data.sub(/\A\[/, ""))
@@ -55,7 +55,7 @@ module Kimurai
       def save_to_pretty_json(item)
         data = JSON.pretty_generate([item])
 
-        if append || @index > 1
+        if @index > 1 || append && File.exists?(path)
           file_content = File.read(path).sub(/\}\n\]\Z/, "\}\,\n")
           File.open(path, "w") do |f|
             f.write(file_content + data.sub(/\A\[\n/, ""))
@@ -68,7 +68,7 @@ module Kimurai
       def save_to_jsonlines(item)
         data = JSON.generate(item)
 
-        if append || @index > 1
+        if @index > 1 || append && File.exists?(path)
           File.open(path, "a") { |file| file.write("\n" + data) }
         else
           File.open(path, "w") { |file| file.write(data) }
@@ -78,7 +78,7 @@ module Kimurai
       def save_to_csv(item)
         data = flatten_hash(item)
 
-        if append || @index > 1
+        if @index > 1 || append && File.exists?(path)
           CSV.open(path, "a+", force_quotes: true) do |csv|
             csv << data.values
           end

data/lib/kimurai/base/storage.rb

@@ -40,11 +40,21 @@ module Kimurai
           if path
             database.transaction do
               database[scope] ||= []
-              database[scope].push(value) unless database[scope].include?(value)
+              if value.class == Array
+                database[scope] += value
+                database[scope].uniq!
+              else
+                database[scope].push(value) unless database[scope].include?(value)
+              end
             end
           else
             database[scope] ||= []
-            database[scope].push(value) unless database[scope].include?(value)
+            if value.class == Array
+              database[scope] += value
+              database[scope].uniq!
+            else
+              database[scope].push(value) unless database[scope].include?(value)
+            end
           end
         end
       end

data/lib/kimurai/browser_builder/mechanize_builder.rb

@@ -80,9 +80,15 @@ module Kimurai
         end
 
         # Browser instance options
+        # skip_request_errors
+        if skip_errors = @config[:skip_request_errors].presence
+          @browser.config.skip_request_errors = skip_errors
+          logger.debug "BrowserBuilder (mechanize): enabled skip_request_errors"
+        end
+
         # retry_request_errors
-        if errors = @config[:retry_request_errors].presence
-          @browser.config.retry_request_errors = errors
+        if retry_errors = @config[:retry_request_errors].presence
+          @browser.config.retry_request_errors = retry_errors
           logger.debug "BrowserBuilder (mechanize): enabled retry_request_errors"
         end
 

data/lib/kimurai/browser_builder/poltergeist_phantomjs_builder.rb

@@ -91,9 +91,15 @@ module Kimurai
         end
 
         # Browser instance options
+        # skip_request_errors
+        if skip_errors = @config[:skip_request_errors].presence
+          @browser.config.skip_request_errors = skip_errors
+          logger.debug "BrowserBuilder (poltergeist_phantomjs): enabled skip_request_errors"
+        end
+
         # retry_request_errors
-        if errors = @config[:retry_request_errors].presence
-          @browser.config.retry_request_errors = errors
+        if retry_errors = @config[:retry_request_errors].presence
+          @browser.config.retry_request_errors = retry_errors
           logger.debug "BrowserBuilder (poltergeist_phantomjs): enabled retry_request_errors"
         end
 

data/lib/kimurai/browser_builder/selenium_chrome_builder.rb

@@ -23,8 +23,15 @@ module Kimurai
         # Register driver
         Capybara.register_driver :selenium_chrome do |app|
           # Create driver options
-          default_args = %w[--disable-gpu --no-sandbox --disable-translate]
-          driver_options = Selenium::WebDriver::Chrome::Options.new(args: default_args)
+          opts = { args: %w[--disable-gpu --no-sandbox --disable-translate] }
+
+          # Provide custom chrome browser path:
+          if chrome_path = Kimurai.configuration.selenium_chrome_path
+            opts.merge!(binary: chrome_path)
+          end
+
+          # See all options here: https://seleniumhq.github.io/selenium/docs/api/rb/Selenium/WebDriver/Chrome/Options.html
+          driver_options = Selenium::WebDriver::Chrome::Options.new(opts)
 
           # Window size
           if size = @config[:window_size].presence
@@ -99,7 +106,8 @@ module Kimurai
             end
           end
 
-          Capybara::Selenium::Driver.new(app, browser: :chrome, options: driver_options, driver_path: "/usr/local/bin/chromedriver")
+          chromedriver_path = Kimurai.configuration.chromedriver_path || "/usr/local/bin/chromedriver"
+          Capybara::Selenium::Driver.new(app, browser: :chrome, options: driver_options, driver_path: chromedriver_path)
         end
 
         # Create browser instance (Capybara session)
@@ -118,9 +126,15 @@ module Kimurai
         end
 
         # Browser instance options
+        # skip_request_errors
+        if skip_errors = @config[:skip_request_errors].presence
+          @browser.config.skip_request_errors = skip_errors
+          logger.debug "BrowserBuilder (selenium_chrome): enabled skip_request_errors"
+        end
+
         # retry_request_errors
-        if errors = @config[:retry_request_errors].presence
-          @browser.config.retry_request_errors = errors
+        if retry_errors = @config[:retry_request_errors].presence
+          @browser.config.retry_request_errors = retry_errors
           logger.debug "BrowserBuilder (selenium_chrome): enabled retry_request_errors"
         end
 

data/lib/kimurai/browser_builder/selenium_firefox_builder.rb

@@ -131,9 +131,15 @@ module Kimurai
         end
 
         # Browser instance options
+        # skip_request_errors
+        if skip_errors = @config[:skip_request_errors].presence
+          @browser.config.skip_request_errors = skip_errors
+          logger.debug "BrowserBuilder (selenium_firefox): enabled skip_request_errors"
+        end
+
         # retry_request_errors
-        if errors = @config[:retry_request_errors].presence
-          @browser.config.retry_request_errors = errors
+        if retry_errors = @config[:retry_request_errors].presence
+          @browser.config.retry_request_errors = retry_errors
           logger.debug "BrowserBuilder (selenium_firefox): enabled retry_request_errors"
         end
 

data/lib/kimurai/capybara_ext/session.rb

@@ -1,5 +1,6 @@
 require 'capybara'
 require 'nokogiri'
+require 'json'
 require_relative 'session/config'
 
 module Capybara
@@ -18,21 +19,30 @@ module Capybara
           spider.class.update(:visits, :requests) if spider.with_info
 
           original_visit(visit_uri)
-        rescue *config.retry_request_errors => e
-          logger.error "Browser: request visit error: #{e.inspect}, url: #{visit_uri}"
-          spider.add_event(:requests_errors, e.inspect) if spider.with_info
-
-          if (retries += 1) <= max_retries
-            logger.info "Browser: sleep #{(sleep_interval += 15)} seconds and process retry № #{retries} to the url: #{visit_uri}"
-            sleep sleep_interval and retry
+        rescue => e
+          if match_error?(e, type: :to_skip)
+            logger.error "Browser: skip request error: #{e.inspect}, url: #{visit_uri}"
+            spider.add_event(:requests_errors, e.inspect) if spider.with_info
+            false
+          elsif match_error?(e, type: :to_retry)
+            logger.error "Browser: retry request error: #{e.inspect}, url: #{visit_uri}"
+            spider.add_event(:requests_errors, e.inspect) if spider.with_info
+
+            if (retries += 1) <= max_retries
+              logger.info "Browser: sleep #{(sleep_interval += 15)} seconds and process retry № #{retries} to the url: #{visit_uri}"
+              sleep sleep_interval and retry
+            else
+              logger.error "Browser: all retries (#{retries - 1}) to the url #{visit_uri} are gone"
+              raise e
+            end
           else
-            logger.error "Browser: all retries (#{retries - 1}) to the url `#{visit_uri}` are gone"
             raise e
           end
         else
           driver.responses += 1 and logger.info "Browser: finished get request to: #{visit_uri}"
           spider.class.update(:visits, :responses) if spider.with_info
           driver.visited = true unless driver.visited
+          true
         ensure
           if spider.with_info
             logger.info "Info: visits: requests: #{spider.class.visits[:requests]}, responses: #{spider.class.visits[:responses]}"
@@ -75,8 +85,13 @@ module Capybara
       logger.info "Browser: driver has been restarted: name: #{mode}, pid: #{driver.pid}, port: #{driver.port}"
     end
 
-    def current_response
-      Nokogiri::HTML(body)
+    def current_response(response_type = :html)
+      case response_type
+      when :html
+        Nokogiri::HTML(body)
+      when :json
+        JSON.parse(body)
+      end
     end
 
     ###
@@ -114,6 +129,27 @@ module Capybara
 
     private
 
+    def match_error?(e, type:)
+      errors = (type == :to_retry ? config.retry_request_errors : config.skip_request_errors)
+      if errors.present?
+        errors.any? do |error|
+          if error.class == Hash
+            match = if error[:message].class == Regexp
+              e.message&.match?(error[:message])
+            else
+              e.message&.include?(error[:message])
+            end
+
+            e.class == error[:error] && match
+          else
+            e.class == error
+          end
+        end
+      else
+        false
+      end
+    end
+
     def process_delay(delay)
       interval = (delay.class == Range ? rand(delay) : delay)
       logger.debug "Browser: sleep #{interval.round(2)} #{'second'.pluralize(interval)} before request..."

data/lib/kimurai/capybara_ext/session/config.rb

@@ -1,12 +1,16 @@
 module Capybara
   class SessionConfig
     attr_accessor :cookies, :proxy, :user_agent
-    attr_writer :retry_request_errors
+    attr_writer :retry_request_errors, :skip_request_errors
 
     def retry_request_errors
       @retry_request_errors ||= []
     end
 
+    def skip_request_errors
+      @skip_request_errors ||= []
+    end
+
     def restart_if
       @restart_if ||= {}
     end

data/lib/kimurai/cli.rb

@@ -146,10 +146,23 @@ module Kimurai
       puts VERSION
     end
 
+    desc "dashboard", "Run dashboard"
+    def dashboard
+      raise "Can't find Kimurai project" unless inside_project?
+
+      require './config/boot'
+      if Object.const_defined?("Kimurai::Dashboard")
+        require 'kimurai/dashboard/app'
+        Kimurai::Dashboard::App.run!
+      else
+        raise "Kimurai::Dashboard is not defined"
+      end
+    end
+
     private
 
     def inside_project?
-      Dir.exists? "spiders"
+      Dir.exists?("spiders") && File.exists?("./config/boot.rb")
     end
   end
 end

data/lib/kimurai/core_ext/hash.rb

@@ -0,0 +1,5 @@
+class Hash
+  def deep_merge_excl(second, exclude)
+    self.merge(second.slice(*exclude)).deep_merge(second.except(*exclude))
+  end
+end

data/lib/kimurai/runner.rb

@@ -2,61 +2,43 @@ require 'pmap'
 
 module Kimurai
   class Runner
-    attr_reader :jobs, :spiders
+    attr_reader :jobs, :spiders, :session_info
 
     def initialize(parallel_jobs:)
       @jobs = parallel_jobs
       @spiders = Kimurai.list
+      @start_time = Time.now
 
-      if time_zone = Kimurai.configuration.time_zone
-        Kimurai.time_zone = time_zone
-      end
-    end
-
-    def run!
-      start_time = Time.now
-      run_id = start_time.to_i
-      running_pids = []
-
-      ENV.store("RBCAT_COLORIZER", "false")
-
-      run_info = {
-        id: run_id,
+      @session_info = {
+        id: @start_time.to_i,
         status: :processing,
-        start_time: start_time,
+        start_time: @start_time,
         stop_time: nil,
         environment: Kimurai.env,
-        concurrent_jobs: jobs,
-        spiders: spiders.keys
+        concurrent_jobs: @jobs,
+        spiders: @spiders.keys
       }
 
-      at_exit do
-        # Prevent queue to process new intems while executing at_exit body
-        Thread.list.each { |t| t.kill if t != Thread.main }
-        # Kill currently running spiders
-        running_pids.each { |pid| Process.kill("INT", pid) }
-
-        error = $!
-        stop_time = Time.now
+      if time_zone = Kimurai.configuration.time_zone
+        Kimurai.time_zone = time_zone
+      end
 
-        if error.nil?
-          run_info.merge!(status: :completed, stop_time: stop_time)
-        else
-          run_info.merge!(status: :failed, error: error.inspect, stop_time: stop_time)
-        end
+      ENV.store("SESSION_ID", @start_time.to_i.to_s)
+      ENV.store("RBCAT_COLORIZER", "false")
+    end
 
-        if at_stop_callback = Kimurai.configuration.runner_at_stop_callback
-          at_stop_callback.call(run_info)
-        end
-        puts "<<< Runner: stopped: #{run_info}"
-      end
+    def run!(exception_on_fail: true)
+      running_pids = []
 
-      puts ">>> Runner: started: #{run_info}"
+      puts ">>> Runner: started: #{session_info}"
       if at_start_callback = Kimurai.configuration.runner_at_start_callback
-        at_start_callback.call(run_info)
+        at_start_callback.call(session_info)
       end
 
+      running = true
       spiders.peach_with_index(jobs) do |spider, i|
+        next unless running
+
         spider_name = spider[0]
         puts "> Runner: started spider: #{spider_name}, index: #{i}"
 
@@ -67,6 +49,23 @@ module Kimurai
         running_pids.delete(pid)
         puts "< Runner: stopped spider: #{spider_name}, index: #{i}"
       end
+    rescue StandardError, SignalException, SystemExit => e
+      running = false
+      session_info.merge!(status: :failed, error: e.inspect, stop_time: Time.now)
+      exception_on_fail ? raise(e) : [session_info, e]
+    else
+      session_info.merge!(status: :completed, stop_time: Time.now)
+    ensure
+      running = false
+      Thread.list.each { |t| t.kill if t != Thread.main }
+
+      # Kill currently running spiders (if any, in case of fail)
+      running_pids.each { |pid| Process.kill("INT", pid) }
+
+      if at_stop_callback = Kimurai.configuration.runner_at_stop_callback
+        at_stop_callback.call(session_info)
+      end
+      puts "<<< Runner: stopped: #{session_info}"
     end
   end
 end

data/lib/kimurai/template/config/application.rb

@@ -29,4 +29,9 @@ Kimurai.configure do |config|
   #   json = JSON.pretty_generate(info)
   #   Sender.send_notification("Stopped session: #{json}")
   # end
+
+  # Provide custom chrome binary path (default is any available chrome/chromium in the PATH):
+  # config.selenium_chrome_path = "/usr/bin/chromium-browser"
+  # Provide custom selenium chromedriver path (default is "/usr/local/bin/chromedriver"):
+  # config.chromedriver_path = "/usr/local/bin/chromedriver"
 end

data/lib/kimurai/template/spiders/application_spider.rb

@@ -81,7 +81,23 @@ class ApplicationSpider < Kimurai::Base
     # works for all drivers
     # skip_duplicate_requests: true,
 
-    # Array of errors to retry while processing a request
+    # Automatically skip provided errors while requesting a page.
+    # If raised error matches one of the errors in the list, then this error will be caught,
+    # and request will be skipped.
+    # It is a good idea to skip errors like NotFound(404), etc.
+    # Format: array where elements are error classes or/and hashes. You can use hash format
+    # for more flexibility: `{ error: "RuntimeError", message: "404 => Net::HTTPNotFound" }`.
+    # Provided `message:` will be compared with a full error message using `String#include?`. Also
+    # you can use regex instead: `{ error: "RuntimeError", message: /404|403/ }`.
+    # skip_request_errors: [{ error: RuntimeError, message: "404 => Net::HTTPNotFound" }],
+
+    # Automatically retry provided errors with a few attempts while requesting a page.
+    # If raised error matches one of the errors in the list, then this error will be caught
+    # and the request will be processed again within a delay. There are 3 attempts:
+    # first: delay 15 sec, second: delay 30 sec, third: delay 45 sec.
+    # If after 3 attempts there is still an exception, then the exception will be raised.
+    # It is a good idea to try to retry errros like `ReadTimeout`, `HTTPBadGateway`, etc.
+    # Format: same like for `skip_request_errors` option.
     # retry_request_errors: [Net::ReadTimeout],
 
     # Restart browser if one of the options is true:
@@ -92,6 +108,8 @@ class ApplicationSpider < Kimurai::Base
       # Restart browser if provided requests limit is exceeded (works for all engines)
       # requests_limit: 100
     },
+
+    # Perform several actions before each request:
     before_request: {
       # Change proxy before each request. The `proxy:` option above should be presented
       # and has lambda format. Works only for poltergeist and mechanize engines

data/lib/kimurai/version.rb

@@ -1,3 +1,3 @@
 module Kimurai
-  VERSION = "1.1.0"
+  VERSION = "1.2.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: kimurai
 version: !ruby/object:Gem::Version
-  version: 1.1.0
+  version: 1.2.0
 platform: ruby
 authors:
 - Victor Afanasev
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2018-09-12 00:00:00.000000000 Z
+date: 2018-09-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: thor
@@ -265,7 +265,6 @@ files:
 - ".gitignore"
 - ".travis.yml"
 - CHANGELOG.md
-- CODE_OF_CONDUCT.md
 - Gemfile
 - LICENSE.txt
 - README.md
@@ -301,6 +300,7 @@ files:
 - lib/kimurai/cli/ansible_command_builder.rb
 - lib/kimurai/cli/generator.rb
 - lib/kimurai/core_ext/array.rb
+- lib/kimurai/core_ext/hash.rb
 - lib/kimurai/core_ext/numeric.rb
 - lib/kimurai/core_ext/string.rb
 - lib/kimurai/pipeline.rb

data/CODE_OF_CONDUCT.md

@@ -1,74 +0,0 @@
-# Contributor Covenant Code of Conduct
-
-## Our Pledge
-
-In the interest of fostering an open and welcoming environment, we as
-contributors and maintainers pledge to making participation in our project and
-our community a harassment-free experience for everyone, regardless of age, body
-size, disability, ethnicity, gender identity and expression, level of experience,
-nationality, personal appearance, race, religion, or sexual identity and
-orientation.
-
-## Our Standards
-
-Examples of behavior that contributes to creating a positive environment
-include:
-
-* Using welcoming and inclusive language
-* Being respectful of differing viewpoints and experiences
-* Gracefully accepting constructive criticism
-* Focusing on what is best for the community
-* Showing empathy towards other community members
-
-Examples of unacceptable behavior by participants include:
-
-* The use of sexualized language or imagery and unwelcome sexual attention or
-advances
-* Trolling, insulting/derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or electronic
-  address, without explicit permission
-* Other conduct which could reasonably be considered inappropriate in a
-  professional setting
-
-## Our Responsibilities
-
-Project maintainers are responsible for clarifying the standards of acceptable
-behavior and are expected to take appropriate and fair corrective action in
-response to any instances of unacceptable behavior.
-
-Project maintainers have the right and responsibility to remove, edit, or
-reject comments, commits, code, wiki edits, issues, and other contributions
-that are not aligned to this Code of Conduct, or to ban temporarily or
-permanently any contributor for other behaviors that they deem inappropriate,
-threatening, offensive, or harmful.
-
-## Scope
-
-This Code of Conduct applies both within project spaces and in public spaces
-when an individual is representing the project or its community. Examples of
-representing a project or community include using an official project e-mail
-address, posting via an official social media account, or acting as an appointed
-representative at an online or offline event. Representation of a project may be
-further defined and clarified by project maintainers.
-
-## Enforcement
-
-Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported by contacting the project team at vicfreefly@gmail.com. All
-complaints will be reviewed and investigated and will result in a response that
-is deemed necessary and appropriate to the circumstances. The project team is
-obligated to maintain confidentiality with regard to the reporter of an incident.
-Further details of specific enforcement policies may be posted separately.
-
-Project maintainers who do not follow or enforce the Code of Conduct in good
-faith may face temporary or permanent repercussions as determined by other
-members of the project's leadership.
-
-## Attribution
-
-This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
-available at [http://contributor-covenant.org/version/1/4][version]
-
-[homepage]: http://contributor-covenant.org
-[version]: http://contributor-covenant.org/version/1/4/