rubium 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 124da1f21fad244bbdeb8adee803043896502e4c58964753bbba6ea8083df750
-  data.tar.gz: 13cda3fb2bd4f121700d08f03a8fbc3b15e38b785dca30074afa13121f4592b0
+  metadata.gz: 42ae899603df19fcd01dc7c565aabcc162be29b7e9a12e508ba1bf3b2c26002a
+  data.tar.gz: 3f5ee794cdb712200821de3b9ecd17cbe6edc085b70e1af393f88fab98ef3de1
 SHA512:
-  metadata.gz: e724053ddf9d97bbaf77db2c3647e1988c5bbace1a56b2f55f72afde1758ace0aaabd1f9d1771647361f8a863a6c33c35a8b616a7b8c4bd4f61278a750b33af0
-  data.tar.gz: 8decfacf86c9751c6ae39e61cae9b8fa1b29a3cb769f8ac3e11e15122364173bf0d20e649e058fe150023e1a7ea38ac13fd919d14abddef6fbfa961d64ca6987
+  metadata.gz: 8d36c239dbe64b1693756615fe7aadf111aa71646fa226320bb85c69945f79ae921f29533ed7f6f6d376a78c82ddd22d4e517aaef536732f8bc72b40ba7010fb
+  data.tar.gz: 80a67649a61f32bf80f4d28195f9da07ec3528c801b00a79908a8baa9103577a70ce7a29e374d66fbf7f9ecbfee8b103a2f7c03c88ad4ffd22f827c8162351e2

data/README.md

@@ -1,6 +1,8 @@
 # Rubium
 
-Rubium is a handy wrapper around [chrome_remote](https://github.com/cavalle/chrome_remote) gem. It adds browsers instances handling, and some Capybara-like methods. It is very lightweight (200 lines of code in the main `Rubium::Browser` class for now) and doens't use Selenium or Capybara.
+> **Rubium updated to 0.2.0 version!** Added new options like `set_cookies`, `restart_after`, `urls_blacklist`, `disable_images` and others. Check the readme below:
+
+Rubium is a handy wrapper around [chrome_remote](https://github.com/cavalle/chrome_remote) gem. It adds browsers instances handling, and some Capybara-like methods. It is very lightweight (250 lines of code in the main `Rubium::Browser` class for now) and doens't use Selenium or Capybara. Consider Rubium as a _very simple_ and _basic_ implementation of [Puppeteer](https://github.com/GoogleChrome/puppeteer) in Ruby language.
 
 You can use Rubium as a lightweight alternative to Selenium/Capybara/Watir if you need to perform some operations (like web scraping) using Headless Chromium and Ruby. Of course, the API currently doesn't has a lot of methods to automate browser, but it has the most frequently used and basic ones.
 
@@ -22,6 +24,12 @@ browser.click("some selector")
 # Get current cookies:
 browser.cookies
 
+# Set cookies (Array of hashes):
+browser.set_cookies([
+  { name: "some_cookie_name", value: "some_cookie_value", domain: ".some-cookie-domain.com" },
+  { name: "another_cookie_name", value: "another_cookie_value", domain: ".another-cookie-domain.com" }
+])
+
 # Fill in some field:
 browser.fill_in("some field selector", "Some text")
 
@@ -39,7 +47,8 @@ browser.evaluate_on_new_document(File.read "browser_inject.js")
 # Evaluate JS code expression:
 browser.execute_script("JS code string")
 
-# Access chrome_remote client directly:
+# Access chrome_remote client (instance of ChromeRemote class) directly:
+# See more here: https://github.com/cavalle/chrome_remote#using-the-chromeremote-api
 browser.client
 
 # Close browser:
@@ -49,18 +58,42 @@ browser.close
 browser.restart!
 ```
 
-There are some options which you can provide while creating browser instance:
+**There are some options** which you can provide while creating browser instance:
 
 ```ruby
 browser = Rubium::Browser.new(
-  debugging_port: 9222, # custom debugging port
-  headless: false, # Run browser in normal (not headless) mode
-  user_agent: "Some user agent", # Custom user-agent
-  proxy_server: "http://1.1.1.1:8080", # Set proxy
+  debugging_port: 9222,                  # custom debugging port. Default is any available port.
+  headless: false,                       # Run browser in normal (not headless) mode. Default is headless.
+  window_size: [1600, 900],              # Custom window size. Default is unset.
+  user_agent: "Some user agent",         # Custom user-agent.
+  proxy_server: "http://1.1.1.1:8080",   # Set proxy.
+  extension_code: "Some JS code string", # Inject custom JS code on each page. See above `evaluate_on_new_document`
+  cookies: [],                           # Set custom cookies, see above `set_cookies`
+  restart_after: 25,                     # Automatically restart browser after N processed requests
+  enable_logger: true,                   # Enable logger to log info about processing requests
+  max_timeout: 30,                       # How long to wait (in seconds) until page will be fully loaded. Default 60 sec.
+  urls_blacklist: ["*some-domain.com*"], # Skip all requests which match provided patterns (wildcard allowed).
+  disable_images: true                   # Do not download images.
 )
 ```
 
-You can provide custom Chrome binary path this way:
+Note that for options `user_agent` and `proxy_server` you can provide `lambda` object instead of string:
+
+```ruby
+USER_AGENTS = ["Safari", "Mozilla", "IE", "Chrome"]
+PROXIES = ["http://1.1.1.1:8080", "http://2.2.2.2:8080", "http://3.3.3.3:8080"]
+
+browser = Rubium::Browser.new(
+  user_agent:   -> { USER_AGENTS.sample },
+  proxy_server: -> { PROXIES.sample },
+  restart_after: 25
+)
+```
+
+> What for: Chrome doesn't provide an API to change proxies on the fly (after browser has been started). It is possible to set proxy while starting Chrome instance by providing CLI argument only. On the other hand, Rubium allows you to automatically restart browser (`restart_after` option) after N processed requests. On each restart, if options `user_agent` and/or `proxy_server` has lambda format, then lambda will be called to fetch fresh value. Thus it's possible to rotate proxies/user-agents without any much effort.
+
+
+**You can provide custom Chrome binary** path this way:
 
 ```ruby
 Rubium.configure do |config|

data/lib/rubium/browser.rb

@@ -4,6 +4,7 @@ require 'random-port'
 require 'cliver'
 require 'timeout'
 require 'securerandom'
+require 'logger'
 
 at_exit do
   Rubium::Browser.running_pids.each { |pid| Process.kill("HUP", pid) }
@@ -14,6 +15,7 @@ module Rubium
     class ConfigurationError < StandardError; end
 
     MAX_CONNECT_WAIT_TIME = 2
+    MAX_DEFAULT_TIMEOUT = 60
 
     class << self
       def ports_pool
@@ -25,26 +27,38 @@ module Rubium
       end
     end
 
-    attr_reader  :client, :devtools_url, :pid, :port, :options
+    attr_reader  :client, :devtools_url, :pid, :port, :options, :processed_requests_count, :logger
 
     def initialize(options = {})
       @options = options
+
+      if @options[:enable_logger]
+        @logger = Logger.new(STDOUT)
+        @logger.progname = self.class.to_s
+      end
+
       create_browser
     end
 
     def restart!
+      logger.info "Restarting..." if options[:enable_logger]
+
       close
       create_browser
     end
 
     def close
-      unless closed?
+      if closed?
+        logger.info "Browser already has been closed" if options[:enable_logger]
+      else
         Process.kill("HUP", @pid)
         self.class.running_pids.delete(@pid)
         self.class.ports_pool.release(@port)
 
         FileUtils.rm_rf(@data_dir) if Dir.exist?(@data_dir)
         @closed = true
+
+        logger.info "Closed browser" if options[:enable_logger]
       end
     end
 
@@ -54,14 +68,28 @@ module Rubium
       @closed
     end
 
-    def goto(url, wait: 30)
+    def goto(url, wait: options[:max_timeout] || MAX_DEFAULT_TIMEOUT)
+      logger.info "Started request: #{url}" if options[:enable_logger]
+      if options[:restart_after] && processed_requests_count >= options[:restart_after]
+        restart!
+      end
+
       response = @client.send_cmd "Page.navigate", url: url
 
-      if wait
-        Timeout.timeout(wait) { @client.wait_for "Page.loadEventFired" }
-      else
-        response
+      # By default, after Page.navigate we should wait till page will load completely
+      # using Page.loadEventFired. But on some websites with Ajax navigation, Page.loadEventFired
+      # will stuck forever. In this case you can provide `wait: false` option to skip waiting.
+      if wait != false
+        # https://chromedevtools.github.io/devtools-protocol/tot/Page#event-frameStoppedLoading
+        Timeout.timeout(wait) do
+          @client.wait_for do |event_name, event_params|
+            event_name == "Page.frameStoppedLoading" && event_params["frameId"] == response["frameId"]
+          end
+        end
       end
+
+      @processed_requests_count += 1
+      logger.info "Finished request: #{url}" if options[:enable_logger]
     end
 
     alias_method :visit, :goto
@@ -106,21 +134,21 @@ module Rubium
     end
 
     def click(selector)
-      @client.send_cmd "Runtime.evaluate", expression: <<~JS
+      @client.send_cmd "Runtime.evaluate", expression: <<~js
         document.querySelector("#{selector}").click();
-      JS
+      js
     end
 
     # https://github.com/cyrus-and/chrome-remote-interface/issues/226#issuecomment-320247756
     # https://stackoverflow.com/a/18937620
     def send_key_on(selector, key)
-      @client.send_cmd "Runtime.evaluate", expression: <<~JS
+      @client.send_cmd "Runtime.evaluate", expression: <<~js
         document.querySelector("#{selector}").dispatchEvent(
           new KeyboardEvent("keydown", {
             bubbles: true, cancelable: true, keyCode: #{key}
           })
         );
-      JS
+      js
     end
 
     # https://github.com/GoogleChrome/puppeteer/blob/master/lib/Page.js#L784
@@ -130,15 +158,24 @@ module Rubium
       @client.send_cmd "Page.addScriptToEvaluateOnNewDocument", source: script
     end
 
+    ###
+
     def cookies
       response = @client.send_cmd "Network.getCookies"
       response["cookies"]
     end
 
+    # https://chromedevtools.github.io/devtools-protocol/tot/Network#method-setCookies
+    def set_cookies(cookies)
+      @client.send_cmd "Network.setCookies", cookies: cookies
+    end
+
+    ###
+
     def fill_in(selector, text)
-      execute_script <<~HEREDOC
+      execute_script <<~js
         document.querySelector("#{selector}").value = "#{text}"
-      HEREDOC
+      js
     end
 
     def execute_script(script)
@@ -148,6 +185,8 @@ module Rubium
     private
 
     def create_browser
+      @processed_requests_count = 0
+
       @port = options[:debugging_port] || self.class.ports_pool.acquire
       @data_dir = "/tmp/rubium_profile_#{SecureRandom.hex}"
 
@@ -196,6 +235,26 @@ module Rubium
       @client.send_cmd "Page.enable"
 
       evaluate_on_new_document(options[:extension_code]) if options[:extension_code]
+
+      set_cookies(options[:cookies]) if options[:cookies]
+
+      if options[:urls_blacklist] || options[:disable_images]
+        urls = []
+
+        if options[:urls_blacklist]
+          urls += options[:urls_blacklist]
+        end
+
+        if options[:disable_images]
+          urls += %w(jpg jpeg png gif swf svg tif).map { |ext| ["*.#{ext}", "*.#{ext}?*"] }.flatten
+          urls << "data:image*"
+        end
+
+        @client.send_cmd "Network.setBlockedURLs", urls: urls
+      end
+
+
+      logger.info "Opened browser" if options[:enable_logger]
     end
 
     def convert_proxy(proxy_string)

data/lib/rubium/version.rb

@@ -1,3 +1,3 @@
 module Rubium
-  VERSION = "0.1.0"
+  VERSION = "0.2.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: rubium
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - Victor Afanasev
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-12-19 00:00:00.000000000 Z
+date: 2019-01-05 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: chrome_remote