ferrum 0.7 → 0.8

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 34f864b5679986d8580fee118735ea2bf62b5b35f1d2ae9f403cdc08168d1d48
-  data.tar.gz: 5899a23fdf4219dfd7aa70b7f1d72d0dcee167a6102b07a5062c36b8c4588f90
+  metadata.gz: 1fb53f7db2767597ff5d6e7ff08536562edec30f5d68fbb2dcac7bf9943e056d
+  data.tar.gz: 46f9f75de5f4c0a4531dbbeda25c538350c3daa167f5de53c9248dbfc6958e34
 SHA512:
-  metadata.gz: 3732e2ba120edbd7e1d28759d6d9fe75c4416f41cc08a535e63751324faec35117a805de3dfdb1467fa3aa55c2885b8c7058d1fce46bb0b69e642210f89bac26
-  data.tar.gz: 426b53078a92ddc04187250335c200dd35dac489eb2fe887f07a7ae8bb07f0652588d278ef18d7389221f692ae3b55488cedfcec0ea8511b57d9ebc9776060db
+  metadata.gz: 0e15da4027c44cd38e3d4eaf6f3384085fbb85fe5760ade585bbaaf2c4ea4fa79db98c23fee3099b33e1d09ab50e1ce8b979ec8848203a3d0c4b45fbe6208b7f
+  data.tar.gz: 7038d36ca9cca2204d9a489380ae0db7d75f202921a3b31b4bd9cbed3c54fb4bd187d5270052dfcdeba300ba615140b33660982dfaa6f99759a3463c52434fc1

data/README.md

@@ -1,57 +1,81 @@
-# Ferrum - fearless Ruby Chrome driver
+# Ferrum - high-level API to control Chrome in Ruby
 
-[![Build Status](https://travis-ci.org/route/ferrum.svg?branch=master)](https://travis-ci.org/route/ferrum)
+[![Build Status](https://travis-ci.org/rubycdp/ferrum.svg?branch=master)](https://travis-ci.org/rubycdp/ferrum)
 
-<img align="right" width="95" height="95"
+<img align="right"
+     width="320" height="241"
      alt="Ferrum logo"
-     src="https://raw.githubusercontent.com/route/ferrum/master/logo.svg?sanitize=true">
+     src="https://raw.githubusercontent.com/rubycdp/ferrum/master/logo.svg?sanitize=true">
 
-As simple as Puppeteer, though even simpler.
+#### As simple as Puppeteer, though even simpler.
 
-It is Ruby clean and high-level API to Chrome. Runs headless by default,
-but you can configure it to run in a non-headless mode. All you need is Ruby and
-Chrome/Chromium. Ferrum connects to the browser via DevTools Protocol.
+It is Ruby clean and high-level API to Chrome. Runs headless by default, but you
+can configure it to run in a headful mode. All you need is Ruby and
+[Chrome](https://www.google.com/chrome/) or
+[Chromium](https://www.chromium.org/). Ferrum connects to the browser by [CDP
+protocol](https://chromedevtools.github.io/devtools-protocol/) and  there's _no_
+Selenium/WebDriver/ChromeDriver dependency. The emphasis was made on a raw CDP
+protocol because Chrome allows you to do so many things that are barely
+supported by WebDriver because it should have consistent design with other
+browsers.
 
-[Cuprite](https://github.com/machinio/cuprite) used to have this code inside in
-one form or another but the thing is you don't need Capybara if you are going to
-crawl sites. You crawl, not test. Besides that clean lightweight API to browser
-is what Ruby was missing, so here it comes.
+* [Cuprite](https://github.com/rubycdp/cuprite) is a pure Ruby driver for
+[Capybara](https://github.com/teamcapybara/capybara) based on Ferrum. If you are
+going to crawl sites you better use Ferrum or
+[Vessel](https://github.com/rubycdp/vessel) because you crawl, not test.
 
-[Vessel](https://github.com/route/vessel) high-level web crawling framework
-based on Ferrum.
+* [Vessel](https://github.com/rubycdp/vessel) high-level web crawling framework
+based on Ferrum. It looks like [Scrapy](https://scrapy.org/) except that it uses
+a real browser in order to grab data.
+
+Web design by [Evrone](https://evrone.com/), what else
+[we build with Ruby on Rails](https://evrone.com/ruby), what else
+[we do at Evrone](https://evrone.com/cases#case-studies).
+
+If you like this project, please consider to
+_[become a backer](https://www.patreon.com/rubycdp_ferrum)_ on Patreon.
 
-If you like this project, please consider to _[become a backer](https://www.patreon.com/rferrum)_
-on Patreon.
 
 ## Index
 
-* [Customization](https://github.com/route/ferrum#customization)
-* [Navigation](https://github.com/route/ferrum#navigation)
-* [Finders](https://github.com/route/ferrum#finders)
-* [Screenshots](https://github.com/route/ferrum#screenshots)
-* [Network](https://github.com/route/ferrum#network)
-* [Mouse](https://github.com/route/ferrum#mouse)
-* [Keyboard](https://github.com/route/ferrum#keyboard)
-* [Cookies](https://github.com/route/ferrum#cookies)
-* [Headers](https://github.com/route/ferrum#headers)
-* [JavaScript](https://github.com/route/ferrum#javascript)
-* [Frames](https://github.com/route/ferrum#frames)
-* [Dialog](https://github.com/route/ferrum#dialog)
+* [Install](https://github.com/rubycdp/ferrum#install)
+* [Examples](https://github.com/rubycdp/ferrum#examples)
+* [Docker](https://github.com/rubycdp/ferrum#docker)
+* [Customization](https://github.com/rubycdp/ferrum#customization)
+* [Navigation](https://github.com/rubycdp/ferrum#navigation)
+* [Finders](https://github.com/rubycdp/ferrum#finders)
+* [Screenshots](https://github.com/rubycdp/ferrum#screenshots)
+* [Network](https://github.com/rubycdp/ferrum#network)
+* [Mouse](https://github.com/rubycdp/ferrum#mouse)
+* [Keyboard](https://github.com/rubycdp/ferrum#keyboard)
+* [Cookies](https://github.com/rubycdp/ferrum#cookies)
+* [Headers](https://github.com/rubycdp/ferrum#headers)
+* [JavaScript](https://github.com/rubycdp/ferrum#javascript)
+* [Frames](https://github.com/rubycdp/ferrum#frames)
+* [Frame](https://github.com/rubycdp/ferrum#frame)
+* [Dialog](https://github.com/rubycdp/ferrum#dialog)
+* [Thread safety](https://github.com/rubycdp/ferrum#thread-safety)
+* [License](https://github.com/rubycdp/ferrum#license)
+
 
 ## Install
 
 There's no official Chrome or Chromium package for Linux don't install it this
-way because it either will be outdated or unofficial, both are bad. Download it
-from official [source](https://www.chromium.org/getting-involved/download-chromium).
+way because it's either outdated or unofficial, both are bad. Download it from
+official [source](https://www.chromium.org/getting-involved/download-chromium).
 Chrome binary should be in the `PATH` or `BROWSER_PATH` or you can pass it as an
-option to browser instance `:browser_path`.
+option to browser instance see `:browser_path` in
+[Customization](https://github.com/rubycdp/ferrum#customization).
 
-Add this to your Gemfile:
+Add this to your `Gemfile` and run `bundle install`.
 
 ``` ruby
 gem "ferrum"
 ```
 
+
+## Examples
+
 Navigate to a website and save a screenshot:
 
 ```ruby
@@ -68,7 +92,7 @@ browser = Ferrum::Browser.new
 browser.goto("https://google.com")
 input = browser.at_xpath("//div[@id='searchform']/form//input[@type='text']")
 input.focus.type("Ruby headless driver for Chrome", :Enter)
-browser.at_css("a > h3").text # => "route/ferrum: Ruby Chrome/Chromium driver - GitHub"
+browser.at_css("a > h3").text # => "rubycdp/ferrum: Ruby Chrome/Chromium driver - GitHub"
 browser.quit
 ```
 
@@ -103,7 +127,17 @@ browser.mouse
 browser.quit
 ```
 
-## Customization ##
+
+## Docker
+
+In docker as root you must pass the no-sandbox browser option:
+
+```ruby
+Ferrum::Browser.new(browser_options: { 'no-sandbox': nil })
+```
+
+
+## Customization
 
 You can customize options with the following code in your test setup:
 
@@ -127,7 +161,7 @@ Ferrum::Browser.new(options)
   * `:js_errors` (Boolean) - When true, JavaScript errors get re-raised in Ruby.
   * `:browser_name` (Symbol) - `:chrome` by default, only experimental support
       for `:firefox` for now.
-  * `:browser_path` (String) - Path to chrome binary, you can also set ENV
+  * `:browser_path` (String) - Path to Chrome binary, you can also set ENV
       variable as `BROWSER_PATH=some/path/chrome bundle exec rspec`.
   * `:browser_options` (Hash) - Additional command line options,
       [see them all](https://peter.sh/experiments/chromium-command-line-switches/)
@@ -138,9 +172,9 @@ Ferrum::Browser.new(options)
       browser process will not be spawned.
   * `:process_timeout` (Integer) - How long to wait for the Chrome process to
       respond on startup
-
-
-#### The API below is for master branch and a subject to change before 1.0
+  * `:ws_max_receive_size` (Integer) - How big messages to accept from Chrome
+      over the web socket, in bytes. Defaults to 64MB. Incoming messages larger
+      than this will cause a `Ferrum::DeadBrowserError`.
 
 
 ## Navigation
@@ -186,6 +220,15 @@ browser.goto("https://github.com/")
 browser.refresh
 ```
 
+#### stop
+
+Stop all navigations and loading pending resources on the page
+
+```ruby
+browser.goto("https://github.com/")
+browser.stop
+```
+
 
 ## Finders
 
@@ -558,6 +601,8 @@ Sets given values as cookie
   * :value `String`
   * :domain `String`
   * :expires `Integer`
+  * :samesite `String`
+  * :httponly `Boolean`
 
 ```ruby
 browser.cookies.set(name: "stealth", value: "omg", domain: "google.com") # => true
@@ -686,17 +731,146 @@ browser.evaluate("window.__injected") # => 42
 
 ## Frames
 
-#### frames
-#### main_frame
-#### frame_by
+#### frames : `Array[Frame] | []`
+
+Returns all the frames current page have.
+
+```ruby
+browser.goto("https://www.w3schools.com/tags/tag_frame.asp")
+browser.frames # =>
+# [
+#   #<Ferrum::Frame @id="C6D104CE454A025FBCF22B98DE612B12" @parent_id=nil @name=nil @state=:stopped_loading @execution_id=1>,
+#   #<Ferrum::Frame @id="C09C4E4404314AAEAE85928EAC109A93" @parent_id="C6D104CE454A025FBCF22B98DE612B12" @state=:stopped_loading @execution_id=2>,
+#   #<Ferrum::Frame @id="2E9C7F476ED09D87A42F2FEE3C6FBC3C" @parent_id="C6D104CE454A025FBCF22B98DE612B12" @state=:stopped_loading @execution_id=3>,
+#   ...
+# ]
+```
+
+#### main_frame : `Frame`
+
+Returns page's main frame, the top of the tree and the parent of all frames.
+
+#### frame_by(\*\*options) : `Frame | nil`
+
+Find frame by given options.
+
+* options `Hash`
+  * :id `String` - Unique frame's id that browser provides
+  * :name `String` - Frame's name if there's one
+
+```ruby
+browser.frame_by(id: "C6D104CE454A025FBCF22B98DE612B12")
+```
+
+
+## Frame
+
+#### id : `String`
+
+Frame's unique id.
+
+#### parent_id : `String | nil`
+
+Parent frame id if this one is nested in another one.
+
+#### execution_id : `Integer`
+
+Execution context id which is used by JS, each frame has it's own context in
+which JS evaluates.
+
+#### name : `String | nil`
+
+If frame was given a name it should be here.
+
+#### state : `Symbol | nil`
+
+One of the states frame's in:
+
+* `:started_loading`
+* `:navigated`
+* `:stopped_loading`
+
+#### url : `String`
+
+Returns current frame's location href.
 
-Play around inside given frame
+```ruby
+browser.goto("https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe")
+frame = browser.frames[1]
+frame.url # => https://interactive-examples.mdn.mozilla.net/pages/tabbed/iframe.html
+```
+
+#### title
+
+Returns current frame's title.
+
+```ruby
+browser.goto("https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe")
+frame = browser.frames[1]
+frame.title # => HTML Demo: <iframe>
+```
+
+#### main? : `Boolean`
+
+If current frame is the main frame of the page (top of the tree).
+
+```ruby
+browser.goto("https://www.w3schools.com/tags/tag_frame.asp")
+frame = browser.frame_by(id: "C09C4E4404314AAEAE85928EAC109A93")
+frame.main? # => false
+```
+
+#### current_url : `String`
+
+Returns current frame's top window location href.
+
+```ruby
+browser.goto("https://www.w3schools.com/tags/tag_frame.asp")
+frame = browser.frame_by(id: "C09C4E4404314AAEAE85928EAC109A93")
+frame.current_url # => "https://www.w3schools.com/tags/tag_frame.asp"
+```
+
+#### current_title : `String`
+
+Returns current frame's top window title.
+
+```ruby
+browser.goto("https://www.w3schools.com/tags/tag_frame.asp")
+frame = browser.frame_by(id: "C09C4E4404314AAEAE85928EAC109A93")
+frame.current_title # => "HTML frame tag"
+```
+
+#### body : `String`
+
+Returns current frame's html.
+
+```ruby
+browser.goto("https://www.w3schools.com/tags/tag_frame.asp")
+frame = browser.frame_by(id: "C09C4E4404314AAEAE85928EAC109A93")
+frame.body # => "<html><head></head><body></body></html>"
+```
+
+#### doctype
+
+Returns current frame's doctype.
+
+```ruby
+browser.goto("https://www.w3schools.com/tags/tag_frame.asp")
+browser.main_frame.doctype # => "<!DOCTYPE html>"
+```
+
+#### set_content(html)
+
+Sets a content of a given frame.
+
+  * html `String`
 
 ```ruby
 browser.goto("https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe")
 frame = browser.frames[1]
-puts frame.title # => HTML Demo: <iframe>
-puts frame.url # => https://interactive-examples.mdn.mozilla.net/pages/tabbed/iframe.html
+frame.body # <html lang="en"><head><style>body {transition: opacity ease-in 0.2s; }...
+frame.set_content("<html><head></head><body><p>lol</p></body></html>")
+frame.body # => <html><head></head><body><p>lol</p></body></html>
 ```
 
 
@@ -781,3 +955,27 @@ t2.join
 
 browser.quit
 ```
+
+
+## License
+
+Copyright 2018-2020 Machinio
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/lib/ferrum.rb

@@ -36,6 +36,12 @@ module Ferrum
     end
   end
 
+  class ProcessTimeoutError < Error
+    def initialize(timeout)
+      super("Browser did not produce websocket url within #{timeout} seconds")
+    end
+  end
+
   class DeadBrowserError < Error
     def initialize(message = "Browser is dead or given window is closed")
       super
@@ -72,8 +78,8 @@ module Ferrum
     attr_reader :class_name, :message
 
     def initialize(response)
-      super
       @class_name, @message = response.values_at("className", "description")
+      super(response.merge("message" => @message))
     end
   end
 

data/lib/ferrum/browser.rb

@@ -16,8 +16,9 @@ module Ferrum
     extend Forwardable
     delegate %i[default_context] => :contexts
     delegate %i[targets create_target create_page page pages windows] => :default_context
-    delegate %i[goto back forward refresh reload
-                at_css at_xpath css xpath current_url title body doctype
+    delegate %i[goto back forward refresh reload stop
+                at_css at_xpath css xpath current_url current_title url title
+                body doctype set_content
                 headers cookies network
                 mouse keyboard
                 screenshot pdf viewport_size
@@ -28,7 +29,7 @@ module Ferrum
     delegate %i[default_user_agent] => :process
 
     attr_reader :client, :process, :contexts, :logger, :js_errors,
-                :slowmo, :base_url, :options, :window_size
+                :slowmo, :base_url, :options, :window_size, :ws_max_receive_size
     attr_writer :timeout
 
     def initialize(options = nil)
@@ -39,9 +40,10 @@ module Ferrum
       @original_window_size = @window_size
 
       @options = Hash(options.merge(window_size: @window_size))
-      @logger, @timeout = @options.values_at(:logger, :timeout)
+      @logger, @timeout, @ws_max_receive_size =
+        @options.values_at(:logger, :timeout, :ws_max_receive_size)
       @js_errors = @options.fetch(:js_errors, false)
-      @slowmo = @options[:slowmo].to_i
+      @slowmo = @options[:slowmo].to_f
 
       if @options.key?(:base_url)
         self.base_url = @options[:base_url]
@@ -114,7 +116,7 @@ module Ferrum
     def start
       Ferrum.started
       @process = Process.start(@options)
-      @client = Client.new(self, @process.ws_url, 0, false)
+      @client = Client.new(self, @process.ws_url)
       @contexts = Contexts.new(self)
     end
   end

data/lib/ferrum/browser/client.rb

@@ -9,12 +9,11 @@ module Ferrum
     class Client
       INTERRUPTIONS = %w[Fetch.requestPaused Fetch.authRequired].freeze
 
-      def initialize(browser, ws_url, start_id = 0, allow_slowmo = true)
-        @command_id = start_id
-        @pendings = Concurrent::Hash.new
+      def initialize(browser, ws_url, id_starts_with: 0)
         @browser = browser
-        @slowmo = @browser.slowmo if allow_slowmo && @browser.slowmo > 0
-        @ws = WebSocket.new(ws_url, @browser.logger)
+        @command_id = id_starts_with
+        @pendings = Concurrent::Hash.new
+        @ws = WebSocket.new(ws_url, @browser.ws_max_receive_size, @browser.logger)
         @subscriber, @interruptor = Subscriber.build(2)
 
         @thread = Thread.new do
@@ -39,7 +38,6 @@ module Ferrum
         pending = Concurrent::IVar.new
         message = build_message(method, params)
         @pendings[message[:id]] = pending
-        sleep(@slowmo) if @slowmo
         @ws.send_message(message)
         data = pending.value!(@browser.timeout)
         @pendings.delete(message[:id])

data/lib/ferrum/browser/process.rb

@@ -133,8 +133,8 @@ module Ferrum
         end
 
         unless ws_url
-          @logger.puts output if @logger
-          raise "Browser process did not produce websocket url within #{timeout} seconds"
+          @logger.puts(output) if @logger
+          raise ProcessTimeoutError.new(timeout)
         end
       end
 

data/lib/ferrum/browser/web_socket.rb

@@ -11,12 +11,13 @@ module Ferrum
 
       attr_reader :url, :messages
 
-      def initialize(url, logger)
+      def initialize(url, max_receive_size, logger)
         @url      = url
         @logger   = logger
         uri       = URI.parse(@url)
         @sock     = TCPSocket.new(uri.host, uri.port)
-        @driver   = ::WebSocket::Driver.client(self)
+        max_receive_size ||= ::WebSocket::Driver::MAX_LENGTH
+        @driver   = ::WebSocket::Driver.client(self, max_length: max_receive_size)
         @messages = Queue.new
 
         @driver.on(:open,    &method(:on_open))
@@ -25,7 +26,9 @@ module Ferrum
 
         @thread = Thread.new do
           Thread.current.abort_on_exception = true
-          Thread.current.report_on_exception = true if Thread.current.respond_to?(:report_on_exception=)
+          if Thread.current.respond_to?(:report_on_exception=)
+            Thread.current.report_on_exception = true
+          end
 
           begin
             while data = @sock.readpartial(512)

data/lib/ferrum/cookies.rb

@@ -23,6 +23,10 @@ module Ferrum
         @attributes["path"]
       end
 
+      def samesite
+        @attributes["sameSite"]
+      end
+
       def size
         @attributes["size"]
       end
@@ -65,6 +69,9 @@ module Ferrum
       cookie[:value]  ||= value
       cookie[:domain] ||= default_domain
 
+      cookie[:httpOnly] = cookie.delete(:httponly) if cookie.key?(:httponly)
+      cookie[:sameSite] = cookie.delete(:samesite) if cookie.key?(:samesite)
+
       expires = cookie.delete(:expires).to_i
       cookie[:expires] = expires if expires > 0
 

data/lib/ferrum/dialog.rb

@@ -14,11 +14,11 @@ module Ferrum
       options = { accept: true }
       response = prompt_text || default_prompt
       options.merge!(promptText: response) if response
-      @page.command("Page.handleJavaScriptDialog", **options)
+      @page.command("Page.handleJavaScriptDialog", slowmoable: true, **options)
     end
 
     def dismiss
-      @page.command("Page.handleJavaScriptDialog", accept: false)
+      @page.command("Page.handleJavaScriptDialog", slowmoable: true, accept: false)
     end
 
     def match?(regexp)

data/lib/ferrum/frame.rb

@@ -7,9 +7,8 @@ module Ferrum
   class Frame
     include DOM, Runtime
 
-    attr_reader :id, :page, :parent_id, :state
-    attr_writer :execution_id
-    attr_accessor :name
+    attr_reader :page, :parent_id, :state
+    attr_accessor :id, :name
 
     def initialize(id, page, parent_id = nil)
       @id, @page, @parent_id = id, page, parent_id
@@ -35,6 +34,15 @@ module Ferrum
       @parent_id.nil?
     end
 
+    def set_content(html)
+      evaluate_async(%(
+        document.open();
+        document.write(arguments[0]);
+        document.close();
+        arguments[1](true);
+      ), @page.timeout, html)
+    end
+
     def execution_id?(execution_id)
       @execution_id == execution_id
     end
@@ -47,6 +55,14 @@ module Ferrum
       @page.event.wait(@page.timeout) ? retry : raise
     end
 
+    def set_execution_id(value)
+      @execution_id ||= value
+    end
+
+    def reset_execution_id
+      @execution_id = nil
+    end
+
     def inspect
       %(#<#{self.class} @id=#{@id.inspect} @parent_id=#{@parent_id.inspect} @name=#{@name.inspect} @state=#{@state.inspect} @execution_id=#{@execution_id.inspect}>)
     end

data/lib/ferrum/frame/dom.rb

@@ -29,65 +29,62 @@ module Ferrum
       end
 
       def doctype
-        evaluate("new XMLSerializer().serializeToString(document.doctype)")
+        evaluate("document.doctype && new XMLSerializer().serializeToString(document.doctype)")
       end
 
       def body
         evaluate("document.documentElement.outerHTML")
       end
 
-      def at_xpath(selector, within: nil)
-        xpath(selector, within: within).first
-      end
-
-      # FIXME: Check within
       def xpath(selector, within: nil)
-        evaluate_async(%(
-          try {
-            let selector = arguments[0];
-            let within = arguments[1] || document;
-            let results = [];
+        code = <<~JS
+          let selector = arguments[0];
+          let within = arguments[1] || document;
+          let results = [];
 
-            let xpath = document.evaluate(selector, within, null, XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null);
-            for (let i = 0; i < xpath.snapshotLength; i++) {
-              results.push(xpath.snapshotItem(i));
-            }
+          let xpath = document.evaluate(selector, within, null, XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null);
+          for (let i = 0; i < xpath.snapshotLength; i++) {
+            results.push(xpath.snapshotItem(i));
+          }
 
-            arguments[2](results);
-          } catch (error) {
-            // DOMException.INVALID_EXPRESSION_ERR is undefined, using pure code
-            if (error.code == DOMException.SYNTAX_ERR || error.code == 51) {
-              throw "Invalid Selector";
-            } else {
-              throw error;
-            }
-          }), @page.timeout, selector, within)
+          arguments[2](results);
+        JS
+
+        evaluate_async(code, @page.timeout, selector, within)
       end
 
-      # FIXME css doesn't work for a frame w/o execution_id
-      def css(selector, within: nil)
-        node_id = within&.node_id || @page.document_id
+      def at_xpath(selector, within: nil)
+        code = <<~JS
+          let selector = arguments[0];
+          let within = arguments[1] || document;
+          let xpath = document.evaluate(selector, within, null, XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null);
+          let result = xpath.snapshotItem(0);
+          arguments[2](result);
+        JS
 
-        ids = @page.command("DOM.querySelectorAll",
-                            nodeId: node_id,
-                            selector: selector)["nodeIds"]
-        ids.map { |id| build_node(id) }.compact
+        evaluate_async(code, @page.timeout, selector, within)
       end
 
-      def at_css(selector, within: nil)
-        node_id = within&.node_id || @page.document_id
+      def css(selector, within: nil)
+        code = <<~JS
+          let selector = arguments[0];
+          let within = arguments[1] || document;
+          let results = within.querySelectorAll(selector);
+          arguments[2](results);
+        JS
 
-        id = @page.command("DOM.querySelector",
-                     nodeId: node_id,
-                     selector: selector)["nodeId"]
-        build_node(id)
+        evaluate_async(code, @page.timeout, selector, within)
       end
 
-      private
+      def at_css(selector, within: nil)
+        code = <<~JS
+          let selector = arguments[0];
+          let within = arguments[1] || document;
+          let result = within.querySelector(selector);
+          arguments[2](result);
+        JS
 
-      def build_node(node_id)
-        description = @page.command("DOM.describeNode", nodeId: node_id)
-        Node.new(self, @page.target_id, node_id, description["node"])
+        evaluate_async(code, @page.timeout, selector, within)
       end
     end
   end

data/lib/ferrum/frame/runtime.rb

@@ -85,8 +85,10 @@ module Ferrum
           options.merge!(returnByValue: by_value)
 
           response = @page.command("Runtime.callFunctionOn",
-                             wait: wait, **options)["result"]
-                            .tap { |r| handle_error(r) }
+                                   wait: wait, slowmoable: true,
+                                   **options)
+          handle_error(response)
+          response = response["result"]
 
           by_value ? response.dig("value") : handle_response(response)
         end
@@ -137,14 +139,18 @@ module Ferrum
           end
 
           response = @page.command("Runtime.callFunctionOn",
-                             **params)["result"].tap { |r| handle_error(r) }
+                                   slowmoable: true,
+                                   **params)
+          handle_error(response)
+          response = response["result"]
 
           handle ? handle_response(response) : response
         end
       end
 
       # FIXME: We should have a central place to handle all type of errors
-      def handle_error(result)
+      def handle_error(response)
+        result = response["result"]
         return if result["subtype"] != "error"
 
         case result["description"]
@@ -220,37 +226,38 @@ module Ferrum
       end
 
       def cyclic?(object_id)
-        @page.command("Runtime.callFunctionOn",
-                objectId: object_id,
-                returnByValue: true,
-                functionDeclaration: <<~JS
-                  function() {
-                    if (Array.isArray(this) &&
-                        this.every(e => e instanceof Node)) {
-                      return false;
-                    }
-
-                    const seen = [];
-                    function detectCycle(obj) {
-                      if (typeof obj === "object") {
-                        if (seen.indexOf(obj) !== -1) {
-                          return true;
-                        }
-                        seen.push(obj);
-                        for (let key in obj) {
-                          if (obj.hasOwnProperty(key) && detectCycle(obj[key])) {
-                            return true;
-                          }
-                        }
-                      }
+        @page.command(
+          "Runtime.callFunctionOn",
+          objectId: object_id,
+          returnByValue: true,
+          functionDeclaration: <<~JS
+            function() {
+              if (Array.isArray(this) &&
+                  this.every(e => e instanceof Node)) {
+                return false;
+              }
 
-                      return false;
+              const seen = [];
+              function detectCycle(obj) {
+                if (typeof obj === "object") {
+                  if (seen.indexOf(obj) !== -1) {
+                    return true;
+                  }
+                  seen.push(obj);
+                  for (let key in obj) {
+                    if (obj.hasOwnProperty(key) && detectCycle(obj[key])) {
+                      return true;
                     }
-
-                    return detectCycle(this);
                   }
-                JS
-               )
+                }
+
+                return false;
+              }
+
+              return detectCycle(this);
+            }
+          JS
+        )
       end
     end
   end

data/lib/ferrum/keyboard.rb

@@ -33,13 +33,13 @@ module Ferrum
     def down(key)
       key = normalize_keys(Array(key))
       type = key[:text] ? "keyDown" : "rawKeyDown"
-      @page.command("Input.dispatchKeyEvent", type: type, **key)
+      @page.command("Input.dispatchKeyEvent", slowmoable: true, type: type, **key)
       self
     end
 
     def up(key)
       key = normalize_keys(Array(key))
-      @page.command("Input.dispatchKeyEvent", type: "keyUp", **key)
+      @page.command("Input.dispatchKeyEvent", slowmoable: true, type: "keyUp", **key)
       self
     end
 
@@ -49,7 +49,7 @@ module Ferrum
       keys.each do |key|
         type = key[:text] ? "keyDown" : "rawKeyDown"
         @page.command("Input.dispatchKeyEvent", type: type, **key)
-        @page.command("Input.dispatchKeyEvent", type: "keyUp", **key)
+        @page.command("Input.dispatchKeyEvent", slowmoable: true, type: "keyUp", **key)
       end
 
       self

data/lib/ferrum/mouse.rb

@@ -32,10 +32,21 @@ module Ferrum
       tap { mouse_event(type: "mouseReleased", **options) }
     end
 
-    # FIXME: steps
     def move(x:, y:, steps: 1)
+      from_x, from_y = @x, @y
       @x, @y = x, y
-      @page.command("Input.dispatchMouseEvent", type: "mouseMoved", x: @x, y: @y)
+
+      steps.times do |i|
+        new_x = from_x + (@x - from_x) * ((i + 1) / steps.to_f)
+        new_y = from_y + (@y - from_y) * ((i + 1) / steps.to_f)
+
+        @page.command("Input.dispatchMouseEvent",
+                      slowmoable: true,
+                      type: "mouseMoved",
+                      x: new_x.to_i,
+                      y: new_y.to_i)
+      end
+
       self
     end
 
@@ -45,7 +56,7 @@ module Ferrum
       button = validate_button(button)
       options = { x: @x, y: @y, type: type, button: button, clickCount: count }
       options.merge!(modifiers: modifiers) if modifiers
-      @page.command("Input.dispatchMouseEvent", wait: wait, **options)
+      @page.command("Input.dispatchMouseEvent", wait: wait, slowmoable: true, **options)
     end
 
     def validate_button(button)

data/lib/ferrum/node.rb

@@ -24,7 +24,7 @@ module Ferrum
     end
 
     def focus
-      tap { page.command("DOM.focus", nodeId: node_id) }
+      tap { page.command("DOM.focus", slowmoable: true, nodeId: node_id) }
     end
 
     def blur
@@ -37,22 +37,23 @@ module Ferrum
 
     # mode: (:left | :right | :double)
     # keys: (:alt, (:ctrl | :control), (:meta | :command), :shift)
-    # offset: { :x, :y }
-    def click(mode: :left, keys: [], offset: {})
-      x, y = find_position(offset[:x], offset[:y])
+    # offset: { :x, :y, :position (:top | :center) }
+    def click(mode: :left, keys: [], offset: {}, delay: 0)
+      x, y = find_position(**offset)
       modifiers = page.keyboard.modifiers(keys)
 
       case mode
       when :right
         page.mouse.move(x: x, y: y)
         page.mouse.down(button: :right, modifiers: modifiers)
+        sleep(delay)
         page.mouse.up(button: :right, modifiers: modifiers)
       when :double
         page.mouse.move(x: x, y: y)
         page.mouse.down(modifiers: modifiers, count: 2)
         page.mouse.up(modifiers: modifiers, count: 2)
       when :left
-        page.mouse.click(x: x, y: y, modifiers: modifiers)
+        page.mouse.click(x: x, y: y, modifiers: modifiers, delay: delay)
       end
 
       self
@@ -63,7 +64,7 @@ module Ferrum
     end
 
     def select_file(value)
-      page.command("DOM.setFileInputFiles", nodeId: node_id, files: Array(value))
+      page.command("DOM.setFileInputFiles", slowmoable: true, nodeId: node_id, files: Array(value))
     end
 
     def at_xpath(selector)
@@ -119,20 +120,31 @@ module Ferrum
       %(#<#{self.class} @target_id=#{@target_id.inspect} @node_id=#{@node_id} @description=#{@description.inspect}>)
     end
 
-    def find_position(offset_x = nil, offset_y = nil)
+    def find_position(x: nil, y: nil, position: :top)
+      offset_x, offset_y = x, y
       quads = get_content_quads
-      offset_x, offset_y = offset_x.to_i, offset_y.to_i
+      x = y = nil
 
-      if offset_x > 0 || offset_y > 0
+      if offset_x && offset_y && position == :top
         point = quads.first
-        [point[:x] + offset_x, point[:y] + offset_y]
+        x = point[:x] + offset_x.to_i
+        y = point[:y] + offset_y.to_i
       else
         x, y = quads.inject([0, 0]) do |memo, point|
           [memo[0] + point[:x],
            memo[1] + point[:y]]
         end
-        [x / 4, y / 4]
+
+        x = x / 4
+        y = y / 4
+      end
+
+      if offset_x && offset_y && position == :center
+        x = x + offset_x.to_i
+        y = y + offset_y.to_i
       end
+
+      [x, y]
     end
 
     private

data/lib/ferrum/page.rb

@@ -31,7 +31,7 @@ module Ferrum
 
     extend Forwardable
     delegate %i[at_css at_xpath css xpath
-                current_url current_title url title body doctype
+                current_url current_title url title body doctype set_content
                 execution_id evaluate evaluate_on evaluate_async execute
                 add_script_tag add_style_tag] => :main_frame
 
@@ -44,13 +44,14 @@ module Ferrum
 
     def initialize(target_id, browser)
       @frames = {}
+      @main_frame = Frame.new(nil, self) 
       @target_id, @browser = target_id, browser
       @event = Event.new.tap(&:set)
 
       host = @browser.process.host
       port = @browser.process.port
       ws_url = "ws://#{host}:#{port}/devtools/page/#{@target_id}"
-      @client = Browser::Client.new(browser, ws_url, 1000)
+      @client = Browser::Client.new(browser, ws_url, id_starts_with: 1000)
 
       @mouse, @keyboard = Mouse.new(self), Keyboard.new(self)
       @headers, @cookies = Headers.new(self), Cookies.new(self)
@@ -99,7 +100,8 @@ module Ferrum
         @browser.command("Browser.setWindowBounds", windowId: @window_id, bounds: { width: width, height: height, windowState: "normal" })
       end
 
-      command("Emulation.setDeviceMetricsOverride", width: width,
+      command("Emulation.setDeviceMetricsOverride", slowmoable: true,
+                                                    width: width,
                                                     height: height,
                                                     deviceScaleFactor: 1,
                                                     mobile: false,
@@ -107,10 +109,14 @@ module Ferrum
     end
 
     def refresh
-      command("Page.reload", wait: timeout)
+      command("Page.reload", wait: timeout, slowmoable: true)
     end
     alias_method :reload, :refresh
 
+    def stop
+      command("Page.stopLoading", slowmoable: true)
+    end
+
     def back
       history_navigate(delta: -1)
     end
@@ -125,9 +131,11 @@ module Ferrum
       enabled
     end
 
-    def command(method, wait: 0, **params)
+    def command(method, wait: 0, slowmoable: false, **params)
       iteration = @event.reset if wait > 0
+      sleep(@browser.slowmo) if slowmoable && @browser.slowmo > 0
       result = @client.command(method, params)
+
       if wait > 0
         @event.wait(wait) # Wait a bit after command and check if iteration has
                           # changed which means there was some network event for
@@ -236,7 +244,9 @@ module Ferrum
 
       if entry = entries[index + delta]
         # Potential wait because of network event
-        command("Page.navigateToHistoryEntry", wait: Mouse::CLICK_WAIT, entryId: entry["id"])
+        command("Page.navigateToHistoryEntry", wait: Mouse::CLICK_WAIT,
+                                               slowmoable: true,
+                                               entryId: entry["id"])
       end
     end
 

data/lib/ferrum/page/frames.rb

@@ -73,21 +73,27 @@ module Ferrum
         on("Runtime.executionContextCreated") do |params|
           context_id = params.dig("context", "id")
           frame_id = params.dig("context", "auxData", "frameId")
+
+          unless @main_frame.id
+            @main_frame.id = frame_id
+            @frames[frame_id] = @main_frame
+          end
+
           frame = @frames[frame_id] || Frame.new(frame_id, self)
-          frame.execution_id = context_id
+          frame.set_execution_id(context_id)
 
-          @main_frame ||= frame
           @frames[frame_id] ||= frame
         end
 
         on("Runtime.executionContextDestroyed") do |params|
           execution_id = params["executionContextId"]
           frame = frames.find { |f| f.execution_id?(execution_id) }
-          frame.execution_id = nil
+          frame.reset_execution_id
         end
 
         on("Runtime.executionContextsCleared") do
           @frames.delete_if { |_, f| !f.main? }
+          @main_frame.reset_execution_id
         end
       end
 

data/lib/ferrum/page/screenshot.rb

@@ -48,8 +48,8 @@ module Ferrum
 
       def document_size
         evaluate <<~JS
-          [document.documentElement.offsetWidth,
-           document.documentElement.offsetHeight]
+          [document.documentElement.scrollWidth,
+           document.documentElement.scrollHeight]
         JS
       end
 

data/lib/ferrum/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Ferrum
-  VERSION = "0.7"
+  VERSION = "0.8"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: ferrum
 version: !ruby/object:Gem::Version
-  version: '0.7'
+  version: '0.8'
 platform: ruby
 authors:
 - Dmitry Vorotilin
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-01-28 00:00:00.000000000 Z
+date: 2020-04-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: websocket-driver
@@ -231,7 +231,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.3
+rubygems_version: 3.1.2
 signing_key: 
 specification_version: 4
 summary: Ruby headless Chrome driver