ferrum 0.6 → 0.9

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4f657c088b22a9ff5809b6ecd0d7005cd0c2252645b262e76faee1b6e3af94c8
-  data.tar.gz: d22842188425c753e0a0e113be52e0176e8a7e1a97211c44bf3016e90aa0d1d4
+  metadata.gz: 8b4d6dc7aa1827fbf559e6025b82d29d15ed0e36a89793049266d8049fadabb9
+  data.tar.gz: 6fab0202e85a17971d613db12e37a7ef85325eaf23f718b6801812df565ac64c
 SHA512:
-  metadata.gz: a519e80c6b1450ba85c555d4c5eb547ffb62f129e7873515ba2f5bad4726548289bcc3b4db95932873620e4b645f2f9b7f12c8bd0fdbe0ddada4f0faae080720
-  data.tar.gz: 91a8f95433f9630dd0f2b36c42ea14bf28b49ee9504738c664399eeb7fc3d192bd26c14ae7de707dcaa8273ec65d7b1a8e9bcaeeff4287392a5e1b312ea03b0d
+  metadata.gz: fb109c1b65e73e8d0088734fa004fae3d15650121d01b44dc9086cdb193bb3c7f56c3ad911a2f6de5cd28522231b4e695b67eb515d90afe93b862eb64cfe7050
+  data.tar.gz: a4d5e4c192cbd634c640d86027f3a8faeaf42efcd8fbaa9b0e8c6e81c04aa9921b7f2353eb739c35850363f1ed58f84d5895714f52ecd333d9ebdb50c1de4031

data/README.md

@@ -1,36 +1,81 @@
-# Ferrum - fearless Ruby Chrome driver
+# Ferrum - high-level API to control Chrome in Ruby
 
-[![Build Status](https://travis-ci.org/route/ferrum.svg?branch=master)](https://travis-ci.org/route/ferrum)
+[![Build Status](https://travis-ci.org/rubycdp/ferrum.svg?branch=master)](https://travis-ci.org/rubycdp/ferrum)
 
-<img align="right" width="95" height="95"
+<img align="right"
+     width="320" height="241"
      alt="Ferrum logo"
-     src="https://raw.githubusercontent.com/route/ferrum/master/logo.svg?sanitize=true">
+     src="https://raw.githubusercontent.com/rubycdp/ferrum/master/logo.svg?sanitize=true">
+
+#### As simple as Puppeteer, though even simpler.
+
+It is Ruby clean and high-level API to Chrome. Runs headless by default, but you
+can configure it to run in a headful mode. All you need is Ruby and
+[Chrome](https://www.google.com/chrome/) or
+[Chromium](https://www.chromium.org/). Ferrum connects to the browser by [CDP
+protocol](https://chromedevtools.github.io/devtools-protocol/) and  there's _no_
+Selenium/WebDriver/ChromeDriver dependency. The emphasis was made on a raw CDP
+protocol because Chrome allows you to do so many things that are barely
+supported by WebDriver because it should have consistent design with other
+browsers.
+
+* [Cuprite](https://github.com/rubycdp/cuprite) is a pure Ruby driver for
+[Capybara](https://github.com/teamcapybara/capybara) based on Ferrum. If you are
+going to crawl sites you better use Ferrum or
+[Vessel](https://github.com/rubycdp/vessel) because you crawl, not test.
+
+* [Vessel](https://github.com/rubycdp/vessel) high-level web crawling framework
+based on Ferrum. It looks like [Scrapy](https://scrapy.org/) except that it uses
+a real browser in order to grab data.
+
+Web design by [Evrone](https://evrone.com/), what else
+[we build with Ruby on Rails](https://evrone.com/ruby), what else
+[we do at Evrone](https://evrone.com/cases#case-studies).
+
+If you like this project, please consider to
+_[become a backer](https://www.patreon.com/rubycdp_ferrum)_ on Patreon.
+
+
+## Index
+
+* [Install](https://github.com/rubycdp/ferrum#install)
+* [Examples](https://github.com/rubycdp/ferrum#examples)
+* [Docker](https://github.com/rubycdp/ferrum#docker)
+* [Customization](https://github.com/rubycdp/ferrum#customization)
+* [Navigation](https://github.com/rubycdp/ferrum#navigation)
+* [Finders](https://github.com/rubycdp/ferrum#finders)
+* [Screenshots](https://github.com/rubycdp/ferrum#screenshots)
+* [Network](https://github.com/rubycdp/ferrum#network)
+* [Mouse](https://github.com/rubycdp/ferrum#mouse)
+* [Keyboard](https://github.com/rubycdp/ferrum#keyboard)
+* [Cookies](https://github.com/rubycdp/ferrum#cookies)
+* [Headers](https://github.com/rubycdp/ferrum#headers)
+* [JavaScript](https://github.com/rubycdp/ferrum#javascript)
+* [Frames](https://github.com/rubycdp/ferrum#frames)
+* [Frame](https://github.com/rubycdp/ferrum#frame)
+* [Dialog](https://github.com/rubycdp/ferrum#dialog)
+* [Thread safety](https://github.com/rubycdp/ferrum#thread-safety)
+* [License](https://github.com/rubycdp/ferrum#license)
 
-As simple as Puppeteer, though even simpler.
-
-It is Ruby clean and high-level API to Chrome. Runs headless by default,
-but you can configure it to run in a non-headless mode. All you need is Ruby and
-Chrome/Chromium. Ferrum connects to the browser via DevTools Protocol.
-
-Relation to [Cuprite](https://github.com/machinio/cuprite). Cuprite used to have
-this code inside in one form or another but the thing is you don't need capybara
-if you are going to crawl sites. You crawl, not test. Besides that clean
-lightweight API to browser is what Ruby was missing, so here it comes.
 
 ## Install
 
 There's no official Chrome or Chromium package for Linux don't install it this
-way because it either will be outdated or unofficial, both are bad. Download it
-from official [source](https://www.chromium.org/getting-involved/download-chromium).
+way because it's either outdated or unofficial, both are bad. Download it from
+official [source](https://www.chromium.org/getting-involved/download-chromium).
 Chrome binary should be in the `PATH` or `BROWSER_PATH` or you can pass it as an
-option to browser instance `:browser_path`.
+option to browser instance see `:browser_path` in
+[Customization](https://github.com/rubycdp/ferrum#customization).
 
-Add this to your Gemfile:
+Add this to your `Gemfile` and run `bundle install`.
 
 ``` ruby
 gem "ferrum"
 ```
 
+
+## Examples
+
 Navigate to a website and save a screenshot:
 
 ```ruby
@@ -45,9 +90,9 @@ Interact with a page:
 ```ruby
 browser = Ferrum::Browser.new
 browser.goto("https://google.com")
-input = browser.at_xpath("//div[@id='searchform']/form//input[@type='text']")
-input.focus.type("Ruby headless driver for Capybara", :Enter)
-browser.at_css("a > h3").text # => "machinio/cuprite: Headless Chrome driver for Capybara - GitHub"
+input = browser.at_xpath("//input[@name='q']")
+input.focus.type("Ruby headless driver for Chrome", :Enter)
+browser.at_css("a > h3").text # => "rubycdp/ferrum: Ruby Chrome/Chromium driver - GitHub"
 browser.quit
 ```
 
@@ -82,7 +127,17 @@ browser.mouse
 browser.quit
 ```
 
-## Customization ##
+
+## Docker
+
+In docker as root you must pass the no-sandbox browser option:
+
+```ruby
+Ferrum::Browser.new(browser_options: { 'no-sandbox': nil })
+```
+
+
+## Customization
 
 You can customize options with the following code in your test setup:
 
@@ -91,31 +146,40 @@ Ferrum::Browser.new(options)
 ```
 
 * options `Hash`
-  * `:browser_path` (String) - Path to chrome binary, you can also set ENV
-      variable as `BROWSER_PATH=some/path/chrome bundle exec rspec`.
   * `:headless` (Boolean) - Set browser as headless or not, `true` by default.
-  * `:slowmo` (Integer | Float) - Set a delay to wait before sending command.
-      Usefull companion of headless option, so that you have time to see changes.
+  * `:xvfb` (Boolean) - Run browser in a virtual framebuffer, `false` by default.
+  * `:window_size` (Array) - The dimensions of the browser window in which to
+      test, expressed as a 2-element array, e.g. [1024, 768]. Default: [1024, 768]
+  * `:extensions` (Array[String | Hash]) - An array of paths to files or JS
+      source code to be preloaded into the browser e.g.:
+      `["/path/to/script.js", { source: "window.secret = 'top'" }]`
   * `:logger` (Object responding to `puts`) - When present, debug output is
       written to this object.
+  * `:slowmo` (Integer | Float) - Set a delay to wait before sending command.
+      Usefull companion of headless option, so that you have time to see changes.
   * `:timeout` (Numeric) - The number of seconds we'll wait for a response when
       communicating with browser. Default is 5.
   * `:js_errors` (Boolean) - When true, JavaScript errors get re-raised in Ruby.
-  * `:window_size` (Array) - The dimensions of the browser window in which to
-      test, expressed as a 2-element array, e.g. [1024, 768]. Default: [1024, 768]
+  * `:browser_name` (Symbol) - `:chrome` by default, only experimental support
+      for `:firefox` for now.
+  * `:browser_path` (String) - Path to Chrome binary, you can also set ENV
+      variable as `BROWSER_PATH=some/path/chrome bundle exec rspec`.
   * `:browser_options` (Hash) - Additional command line options,
       [see them all](https://peter.sh/experiments/chromium-command-line-switches/)
       e.g. `{ "ignore-certificate-errors" => nil }`
-  * `:extensions` (Array) - An array of JS files to be preloaded into the browser
+  * `:ignore_default_browser_options` (Boolean) - Ferrum has a number of default
+      options it passes to the browser, if you set this to `true` then only
+      options you put in `:browser_options` will be passed to the browser,
+      except required ones of course.
   * `:port` (Integer) - Remote debugging port for headless Chrome
   * `:host` (String) - Remote debugging address for headless Chrome
   * `:url` (String) - URL for a running instance of Chrome. If this is set, a
       browser process will not be spawned.
   * `:process_timeout` (Integer) - How long to wait for the Chrome process to
       respond on startup
-
-
-#### The API below is for master branch and a subject to change before 1.0
+  * `:ws_max_receive_size` (Integer) - How big messages to accept from Chrome
+      over the web socket, in bytes. Defaults to 64MB. Incoming messages larger
+      than this will cause a `Ferrum::DeadBrowserError`.
 
 
 ## Navigation
@@ -161,6 +225,15 @@ browser.goto("https://github.com/")
 browser.refresh
 ```
 
+#### stop
+
+Stop all navigations and loading pending resources on the page
+
+```ruby
+browser.goto("https://github.com/")
+browser.stop
+```
+
 
 ## Finders
 
@@ -340,6 +413,24 @@ browser.goto("https://github.com/")
 browser.network.status # => 200
 ```
 
+#### wait_for_idle(\*\*options)
+
+Waits for network idle or raises `Ferrum::TimeoutError` error
+
+* options `Hash`
+  * :connections `Integer` how many connections are allowed for network to be
+    idling, `0` by default
+  * :duration `Float` sleep for given amount of time and check again, `0.05` by
+    default
+  * :timeout `Float` during what time we try to check idle, `browser.timeout`
+    by default
+
+```ruby
+browser.goto("https://example.com/")
+browser.at_xpath("//a[text() = 'No UI changes button']").click
+browser.network.wait_for_idle
+```
+
 #### clear(type)
 
 Clear browser's cache or collected traffic.
@@ -515,6 +606,8 @@ Sets given values as cookie
   * :value `String`
   * :domain `String`
   * :expires `Integer`
+  * :samesite `String`
+  * :httponly `Boolean`
 
 ```ruby
 browser.cookies.set(name: "stealth", value: "omg", domain: "google.com") # => true
@@ -626,22 +719,163 @@ browser.add_script_tag(url: "http://example.com/stylesheet.css") # => true
 
 ```ruby
 browser.add_style_tag(content: "h1 { font-size: 40px; }") # => true
+
+```
+#### bypass_csp(enabled) : `Boolean`
+
+* enabled `Boolean`, `true` by default
+
+```ruby
+browser.bypass_csp # => true
+browser.goto("https://github.com/ruby-concurrency/concurrent-ruby/blob/master/docs-source/promises.in.md")
+browser.refresh
+browser.add_script_tag(content: "window.__injected = 42")
+browser.evaluate("window.__injected") # => 42
 ```
 
 
 ## Frames
 
-#### frames
-#### main_frame
-#### frame_by
+#### frames : `Array[Frame] | []`
+
+Returns all the frames current page have.
+
+```ruby
+browser.goto("https://www.w3schools.com/tags/tag_frame.asp")
+browser.frames # =>
+# [
+#   #<Ferrum::Frame @id="C6D104CE454A025FBCF22B98DE612B12" @parent_id=nil @name=nil @state=:stopped_loading @execution_id=1>,
+#   #<Ferrum::Frame @id="C09C4E4404314AAEAE85928EAC109A93" @parent_id="C6D104CE454A025FBCF22B98DE612B12" @state=:stopped_loading @execution_id=2>,
+#   #<Ferrum::Frame @id="2E9C7F476ED09D87A42F2FEE3C6FBC3C" @parent_id="C6D104CE454A025FBCF22B98DE612B12" @state=:stopped_loading @execution_id=3>,
+#   ...
+# ]
+```
+
+#### main_frame : `Frame`
+
+Returns page's main frame, the top of the tree and the parent of all frames.
+
+#### frame_by(\*\*options) : `Frame | nil`
+
+Find frame by given options.
+
+* options `Hash`
+  * :id `String` - Unique frame's id that browser provides
+  * :name `String` - Frame's name if there's one
+
+```ruby
+browser.frame_by(id: "C6D104CE454A025FBCF22B98DE612B12")
+```
+
+
+## Frame
+
+#### id : `String`
+
+Frame's unique id.
 
-Play around inside given frame
+#### parent_id : `String | nil`
+
+Parent frame id if this one is nested in another one.
+
+#### execution_id : `Integer`
+
+Execution context id which is used by JS, each frame has it's own context in
+which JS evaluates.
+
+#### name : `String | nil`
+
+If frame was given a name it should be here.
+
+#### state : `Symbol | nil`
+
+One of the states frame's in:
+
+* `:started_loading`
+* `:navigated`
+* `:stopped_loading`
+
+#### url : `String`
+
+Returns current frame's location href.
+
+```ruby
+browser.goto("https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe")
+frame = browser.frames[1]
+frame.url # => https://interactive-examples.mdn.mozilla.net/pages/tabbed/iframe.html
+```
+
+#### title
+
+Returns current frame's title.
 
 ```ruby
 browser.goto("https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe")
 frame = browser.frames[1]
-puts frame.title # => HTML Demo: <iframe>
-puts frame.url # => https://interactive-examples.mdn.mozilla.net/pages/tabbed/iframe.html
+frame.title # => HTML Demo: <iframe>
+```
+
+#### main? : `Boolean`
+
+If current frame is the main frame of the page (top of the tree).
+
+```ruby
+browser.goto("https://www.w3schools.com/tags/tag_frame.asp")
+frame = browser.frame_by(id: "C09C4E4404314AAEAE85928EAC109A93")
+frame.main? # => false
+```
+
+#### current_url : `String`
+
+Returns current frame's top window location href.
+
+```ruby
+browser.goto("https://www.w3schools.com/tags/tag_frame.asp")
+frame = browser.frame_by(id: "C09C4E4404314AAEAE85928EAC109A93")
+frame.current_url # => "https://www.w3schools.com/tags/tag_frame.asp"
+```
+
+#### current_title : `String`
+
+Returns current frame's top window title.
+
+```ruby
+browser.goto("https://www.w3schools.com/tags/tag_frame.asp")
+frame = browser.frame_by(id: "C09C4E4404314AAEAE85928EAC109A93")
+frame.current_title # => "HTML frame tag"
+```
+
+#### body : `String`
+
+Returns current frame's html.
+
+```ruby
+browser.goto("https://www.w3schools.com/tags/tag_frame.asp")
+frame = browser.frame_by(id: "C09C4E4404314AAEAE85928EAC109A93")
+frame.body # => "<html><head></head><body></body></html>"
+```
+
+#### doctype
+
+Returns current frame's doctype.
+
+```ruby
+browser.goto("https://www.w3schools.com/tags/tag_frame.asp")
+browser.main_frame.doctype # => "<!DOCTYPE html>"
+```
+
+#### set_content(html)
+
+Sets a content of a given frame.
+
+  * html `String`
+
+```ruby
+browser.goto("https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe")
+frame = browser.frames[1]
+frame.body # <html lang="en"><head><style>body {transition: opacity ease-in 0.2s; }...
+frame.set_content("<html><head></head><body><p>lol</p></body></html>")
+frame.body # => <html><head></head><body><p>lol</p></body></html>
 ```
 
 
@@ -726,3 +960,27 @@ t2.join
 
 browser.quit
 ```
+
+
+## License
+
+Copyright 2018-2020 Machinio
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/lib/ferrum.rb

@@ -10,8 +10,14 @@ module Ferrum
   class NotImplementedError < Error; end
 
   class StatusError < Error
-    def initialize(url)
-      super("Request to #{url} failed to reach server, check DNS and/or server status")
+    def initialize(url, pendings = [])
+      message = if pendings.empty?
+                  "Request to #{url} failed to reach server, check DNS and/or server status"
+                else
+                  "Request to #{url} reached server, but there are still pending connections: #{pendings.join(', ')}"
+                end
+
+      super(message)
     end
   end
 
@@ -30,12 +36,31 @@ module Ferrum
     end
   end
 
+  class ProcessTimeoutError < Error
+    def initialize(timeout)
+      super("Browser did not produce websocket url within #{timeout} seconds")
+    end
+  end
+
   class DeadBrowserError < Error
-    def initialize(message = "Browser is dead")
+    def initialize(message = "Browser is dead or given window is closed")
       super
     end
   end
 
+  class NodeIsMovingError < Error
+    def initialize(node, prev, current)
+      @node, @prev, @current = node, prev, current
+      super(message)
+    end
+
+    def message
+      "#{@node.inspect} that you're trying to click is moving, hence " \
+      "we cannot. Previosuly it was at #{@prev.inspect} but now at " \
+      "#{@current.inspect}."
+    end
+  end
+
   class BrowserError < Error
     attr_reader :response
 
@@ -66,8 +91,8 @@ module Ferrum
     attr_reader :class_name, :message
 
     def initialize(response)
-      super
       @class_name, @message = response.values_at("className", "description")
+      super(response.merge("message" => @message))
     end
   end