kimurai 1.0.0 → 1.0.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d108c41e5da08b22c21cc6c71cc3ac7056ddd1af32054c22a22f0c59658bfcb4
-  data.tar.gz: 8a8d32b7b8646eb50bd9f71d8986edc2ac78efc0e2e6a437b3280cff4418c5dd
+  metadata.gz: 7f2185614ca5aa8486c17e0c43b3b035cf22cd18d51617430f556a12af3dc7c8
+  data.tar.gz: 9e5c296feb5d020aa13bcfaa7f6f4c77d839ff373e8aa9d3e0abcc953aaa89de
 SHA512:
-  metadata.gz: 4c82647cbe276980ef0a246693c7e68c08651351a549f99fbc6618bc9836c4a4ba83b4d09e1e29d06abcfa0d4f70443fb88682f57c544c0218b22940834a48b1
-  data.tar.gz: 845f04c77fbb5e53b24d048e60f23e2c0f9fdeb4d2fde7dcaaa04bebfebc4454777ade03cae895e444583aafb6c8e56038d0d722589fde10076091903646fdf7
+  metadata.gz: 07d92edd8719cbfc701ac7d82975d4c06f5ba9f6adb0bdbbc6731f81655d70d077d140efa54b473b462058042078abab9218f5f00dab244f7478f91c62c8e24b
+  data.tar.gz: 5dc6a70b6379a46c58c917455a7eace96c1093944125888cbc2f9b2af93cf065de0ca00e9d98e786d9a2fbc3a53a1ea3dbf1712e703984d063dfc937ad5e0c71

data/CHANGELOG.md

@@ -0,0 +1,6 @@
+# CHANGELOG
+## HEAD
+
+## 1.0.1
+* Add missing `logger` method to pipeline
+* Fix `set_proxy` in Mechanize and Poltergeist builders

data/README.md

@@ -6,6 +6,18 @@
   <h1>Kimura Framework</h1>
 </div>
 
+> **Note about v1.0.0 version:**
+> * The code was massively refactored for a [support](#using-kimurai-inside-existing-ruby-application) to run spiders multiple times from inside a single process. Now it's possible to run Kimurai spiders using background jobs like Sidekiq.
+> * `require 'kimurai'` doesn't require any gems except Active Support. Only when a particular spider [starts](#crawl-method), Capybara will be required with a specific driver.
+> * Although Kimurai [extends](lib/kimurai/capybara_ext) Capybara (all the magic happens inside [extended](lib/kimurai/capybara_ext/session.rb) `Capybara::Session#visit` method), session instances which were created manually will behave normally.
+> * No spaghetti code with `case/when/end` blocks anymore. All drivers [were extended](lib/kimurai/capybara_ext) to support unified methods for cookies, proxies, headers, etc.
+> * `selenium_url_to_set_cookies` @config option don't need anymore if you're use Selenium-like engine with custom cookies setting.
+> * Small changes in design (check the readme again to see what was changed)
+> * Stats database with a web dashboard were removed
+> * Again, massive refactor. Code now looks much better than it was before.
+
+<br>
+
 Kimurai is a modern web scraping framework written in Ruby which **works out of box with Headless Chromium/Firefox, PhantomJS**, or simple HTTP requests and **allows to scrape and interact with JavaScript rendered websites.**
 
 Kimurai based on well-known [Capybara](https://github.com/teamcapybara/capybara) and [Nokogiri](https://github.com/sparklemotion/nokogiri) gems, so you don't have to learn anything new. Lets see:
@@ -217,7 +229,6 @@ I, [2018-08-22 13:33:30 +0400#23356] [M: 47375890851320]  INFO -- infinite_scrol
 * [Kimurai](#kimurai)
   * [Features](#features)
   * [Table of Contents](#table-of-contents)
-  * [Note about v1.0.0 version](#note-about-v1-0-0-version)
   * [Installation](#installation)
   * [Getting to Know](#getting-to-know)
     * [Interactive console](#interactive-console)
@@ -255,12 +266,6 @@ I, [2018-08-22 13:33:30 +0400#23356] [M: 47375890851320]  INFO -- infinite_scrol
   * [Chat Support and Feedback](#chat-support-and-feedback)
   * [License](#license)
 
-## Note about v1.0.0 version
-* The code was massively refactored for a [support](#using-kimurai-inside-existing-ruby-application) to run spiders multiple times from inside a single process. Now it's possible to run Kimurai spiders using background jobs like Sidekiq.
-* `require 'kimurai'` doesn't require any gems except Active Support. Only when a particular spider [starts](#crawl-method), Capybara will be required with a specific driver.
-* Although Kimurai [extends](lib/kimurai/capybara_ext) Capybara (all the magic happens inside [extended](lib/kimurai/capybara_ext/session.rb) `Capybara::Session#visit` method), session instances which were created manually will behave normally.
-* Small changes in design (check the readme again to see what was changed)
-* Again, massive refactor. Code now looks much better than it was before.
 
 ## Installation
 Kimurai requires Ruby version `>= 2.5.0`. Supported platforms: `Linux` and `Mac OS X`.
@@ -1604,7 +1609,7 @@ To generate a new spider in the project, run:
 
 ```bash
 $ kimurai generate spider example_spider
-      create  crawlers/example_spider.rb
+      create  spiders/example_spider.rb
 ```
 
 Command will generate a new spider class inherited from `ApplicationSpider`:

data/lib/kimurai/browser_builder/mechanize_builder.rb

@@ -37,7 +37,7 @@ module Kimurai
           if type == "socks5"
             logger.error "BrowserBuilder (mechanize): can't set socks5 proxy (not supported), skipped"
           else
-            @browser.set_proxy(*proxy_string.split(":"))
+            @browser.driver.set_proxy(*proxy_string.split(":"))
             logger.debug "BrowserBuilder (mechanize): enabled #{type} proxy, ip: #{ip}, port: #{port}"
           end
         end
@@ -84,7 +84,7 @@ module Kimurai
 
         # restart_if
         if @config.dig(:browser, :restart_if).present?
-          logger.error "BrowserBuilder (mechanize): `browser restart_if` options not supported by Mechanize, skipped"
+          logger.warn "BrowserBuilder (mechanize): `browser restart_if` options not supported by Mechanize, skipped"
         end
 
         # before_request clear_cookies

data/lib/kimurai/browser_builder/poltergeist_phantomjs_builder.rb

@@ -59,7 +59,7 @@ module Kimurai
           proxy_string = (proxy.class == Proc ? proxy.call : proxy).strip
           ip, port, type = proxy_string.split(":")
 
-          @browser.set_proxy(*proxy_string.split(":"))
+          @browser.driver.set_proxy(*proxy_string.split(":"))
           logger.debug "BrowserBuilder (poltergeist_phantomjs): enabled #{type} proxy, ip: #{ip}, port: #{port}"
         end
 

data/lib/kimurai/pipeline.rb

@@ -21,5 +21,9 @@ module Kimurai
     def save_to(path, item, format:, position: true)
       spider.save_to(path, item, format: format, position: position)
     end
+
+    def logger
+      spider.logger
+    end
   end
 end

data/lib/kimurai/version.rb

@@ -1,3 +1,3 @@
 module Kimurai
-  VERSION = "1.0.0"
+  VERSION = "1.0.1"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: kimurai
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.0.1
 platform: ruby
 authors:
 - Victor Afanasev
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2018-08-23 00:00:00.000000000 Z
+date: 2018-08-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: thor
@@ -264,6 +264,7 @@ extra_rdoc_files: []
 files:
 - ".gitignore"
 - ".travis.yml"
+- CHANGELOG.md
 - CODE_OF_CONDUCT.md
 - Gemfile
 - LICENSE.txt