ferrum 0.7 → 0.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 34f864b5679986d8580fee118735ea2bf62b5b35f1d2ae9f403cdc08168d1d48
4
- data.tar.gz: 5899a23fdf4219dfd7aa70b7f1d72d0dcee167a6102b07a5062c36b8c4588f90
3
+ metadata.gz: 1fb53f7db2767597ff5d6e7ff08536562edec30f5d68fbb2dcac7bf9943e056d
4
+ data.tar.gz: 46f9f75de5f4c0a4531dbbeda25c538350c3daa167f5de53c9248dbfc6958e34
5
5
  SHA512:
6
- metadata.gz: 3732e2ba120edbd7e1d28759d6d9fe75c4416f41cc08a535e63751324faec35117a805de3dfdb1467fa3aa55c2885b8c7058d1fce46bb0b69e642210f89bac26
7
- data.tar.gz: 426b53078a92ddc04187250335c200dd35dac489eb2fe887f07a7ae8bb07f0652588d278ef18d7389221f692ae3b55488cedfcec0ea8511b57d9ebc9776060db
6
+ metadata.gz: 0e15da4027c44cd38e3d4eaf6f3384085fbb85fe5760ade585bbaaf2c4ea4fa79db98c23fee3099b33e1d09ab50e1ce8b979ec8848203a3d0c4b45fbe6208b7f
7
+ data.tar.gz: 7038d36ca9cca2204d9a489380ae0db7d75f202921a3b31b4bd9cbed3c54fb4bd187d5270052dfcdeba300ba615140b33660982dfaa6f99759a3463c52434fc1
data/README.md CHANGED
@@ -1,57 +1,81 @@
1
- # Ferrum - fearless Ruby Chrome driver
1
+ # Ferrum - high-level API to control Chrome in Ruby
2
2
 
3
- [![Build Status](https://travis-ci.org/route/ferrum.svg?branch=master)](https://travis-ci.org/route/ferrum)
3
+ [![Build Status](https://travis-ci.org/rubycdp/ferrum.svg?branch=master)](https://travis-ci.org/rubycdp/ferrum)
4
4
 
5
- <img align="right" width="95" height="95"
5
+ <img align="right"
6
+ width="320" height="241"
6
7
  alt="Ferrum logo"
7
- src="https://raw.githubusercontent.com/route/ferrum/master/logo.svg?sanitize=true">
8
+ src="https://raw.githubusercontent.com/rubycdp/ferrum/master/logo.svg?sanitize=true">
8
9
 
9
- As simple as Puppeteer, though even simpler.
10
+ #### As simple as Puppeteer, though even simpler.
10
11
 
11
- It is Ruby clean and high-level API to Chrome. Runs headless by default,
12
- but you can configure it to run in a non-headless mode. All you need is Ruby and
13
- Chrome/Chromium. Ferrum connects to the browser via DevTools Protocol.
12
+ It is Ruby clean and high-level API to Chrome. Runs headless by default, but you
13
+ can configure it to run in a headful mode. All you need is Ruby and
14
+ [Chrome](https://www.google.com/chrome/) or
15
+ [Chromium](https://www.chromium.org/). Ferrum connects to the browser by [CDP
16
+ protocol](https://chromedevtools.github.io/devtools-protocol/) and there's _no_
17
+ Selenium/WebDriver/ChromeDriver dependency. The emphasis was made on a raw CDP
18
+ protocol because Chrome allows you to do so many things that are barely
19
+ supported by WebDriver because it should have consistent design with other
20
+ browsers.
14
21
 
15
- [Cuprite](https://github.com/machinio/cuprite) used to have this code inside in
16
- one form or another but the thing is you don't need Capybara if you are going to
17
- crawl sites. You crawl, not test. Besides that clean lightweight API to browser
18
- is what Ruby was missing, so here it comes.
22
+ * [Cuprite](https://github.com/rubycdp/cuprite) is a pure Ruby driver for
23
+ [Capybara](https://github.com/teamcapybara/capybara) based on Ferrum. If you are
24
+ going to crawl sites you better use Ferrum or
25
+ [Vessel](https://github.com/rubycdp/vessel) because you crawl, not test.
19
26
 
20
- [Vessel](https://github.com/route/vessel) high-level web crawling framework
21
- based on Ferrum.
27
+ * [Vessel](https://github.com/rubycdp/vessel) high-level web crawling framework
28
+ based on Ferrum. It looks like [Scrapy](https://scrapy.org/) except that it uses
29
+ a real browser in order to grab data.
30
+
31
+ Web design by [Evrone](https://evrone.com/), what else
32
+ [we build with Ruby on Rails](https://evrone.com/ruby), what else
33
+ [we do at Evrone](https://evrone.com/cases#case-studies).
34
+
35
+ If you like this project, please consider to
36
+ _[become a backer](https://www.patreon.com/rubycdp_ferrum)_ on Patreon.
22
37
 
23
- If you like this project, please consider to _[become a backer](https://www.patreon.com/rferrum)_
24
- on Patreon.
25
38
 
26
39
  ## Index
27
40
 
28
- * [Customization](https://github.com/route/ferrum#customization)
29
- * [Navigation](https://github.com/route/ferrum#navigation)
30
- * [Finders](https://github.com/route/ferrum#finders)
31
- * [Screenshots](https://github.com/route/ferrum#screenshots)
32
- * [Network](https://github.com/route/ferrum#network)
33
- * [Mouse](https://github.com/route/ferrum#mouse)
34
- * [Keyboard](https://github.com/route/ferrum#keyboard)
35
- * [Cookies](https://github.com/route/ferrum#cookies)
36
- * [Headers](https://github.com/route/ferrum#headers)
37
- * [JavaScript](https://github.com/route/ferrum#javascript)
38
- * [Frames](https://github.com/route/ferrum#frames)
39
- * [Dialog](https://github.com/route/ferrum#dialog)
41
+ * [Install](https://github.com/rubycdp/ferrum#install)
42
+ * [Examples](https://github.com/rubycdp/ferrum#examples)
43
+ * [Docker](https://github.com/rubycdp/ferrum#docker)
44
+ * [Customization](https://github.com/rubycdp/ferrum#customization)
45
+ * [Navigation](https://github.com/rubycdp/ferrum#navigation)
46
+ * [Finders](https://github.com/rubycdp/ferrum#finders)
47
+ * [Screenshots](https://github.com/rubycdp/ferrum#screenshots)
48
+ * [Network](https://github.com/rubycdp/ferrum#network)
49
+ * [Mouse](https://github.com/rubycdp/ferrum#mouse)
50
+ * [Keyboard](https://github.com/rubycdp/ferrum#keyboard)
51
+ * [Cookies](https://github.com/rubycdp/ferrum#cookies)
52
+ * [Headers](https://github.com/rubycdp/ferrum#headers)
53
+ * [JavaScript](https://github.com/rubycdp/ferrum#javascript)
54
+ * [Frames](https://github.com/rubycdp/ferrum#frames)
55
+ * [Frame](https://github.com/rubycdp/ferrum#frame)
56
+ * [Dialog](https://github.com/rubycdp/ferrum#dialog)
57
+ * [Thread safety](https://github.com/rubycdp/ferrum#thread-safety)
58
+ * [License](https://github.com/rubycdp/ferrum#license)
59
+
40
60
 
41
61
  ## Install
42
62
 
43
63
  There's no official Chrome or Chromium package for Linux don't install it this
44
- way because it either will be outdated or unofficial, both are bad. Download it
45
- from official [source](https://www.chromium.org/getting-involved/download-chromium).
64
+ way because it's either outdated or unofficial, both are bad. Download it from
65
+ official [source](https://www.chromium.org/getting-involved/download-chromium).
46
66
  Chrome binary should be in the `PATH` or `BROWSER_PATH` or you can pass it as an
47
- option to browser instance `:browser_path`.
67
+ option to browser instance see `:browser_path` in
68
+ [Customization](https://github.com/rubycdp/ferrum#customization).
48
69
 
49
- Add this to your Gemfile:
70
+ Add this to your `Gemfile` and run `bundle install`.
50
71
 
51
72
  ``` ruby
52
73
  gem "ferrum"
53
74
  ```
54
75
 
76
+
77
+ ## Examples
78
+
55
79
  Navigate to a website and save a screenshot:
56
80
 
57
81
  ```ruby
@@ -68,7 +92,7 @@ browser = Ferrum::Browser.new
68
92
  browser.goto("https://google.com")
69
93
  input = browser.at_xpath("//div[@id='searchform']/form//input[@type='text']")
70
94
  input.focus.type("Ruby headless driver for Chrome", :Enter)
71
- browser.at_css("a > h3").text # => "route/ferrum: Ruby Chrome/Chromium driver - GitHub"
95
+ browser.at_css("a > h3").text # => "rubycdp/ferrum: Ruby Chrome/Chromium driver - GitHub"
72
96
  browser.quit
73
97
  ```
74
98
 
@@ -103,7 +127,17 @@ browser.mouse
103
127
  browser.quit
104
128
  ```
105
129
 
106
- ## Customization ##
130
+
131
+ ## Docker
132
+
133
+ In docker as root you must pass the no-sandbox browser option:
134
+
135
+ ```ruby
136
+ Ferrum::Browser.new(browser_options: { 'no-sandbox': nil })
137
+ ```
138
+
139
+
140
+ ## Customization
107
141
 
108
142
  You can customize options with the following code in your test setup:
109
143
 
@@ -127,7 +161,7 @@ Ferrum::Browser.new(options)
127
161
  * `:js_errors` (Boolean) - When true, JavaScript errors get re-raised in Ruby.
128
162
  * `:browser_name` (Symbol) - `:chrome` by default, only experimental support
129
163
  for `:firefox` for now.
130
- * `:browser_path` (String) - Path to chrome binary, you can also set ENV
164
+ * `:browser_path` (String) - Path to Chrome binary, you can also set ENV
131
165
  variable as `BROWSER_PATH=some/path/chrome bundle exec rspec`.
132
166
  * `:browser_options` (Hash) - Additional command line options,
133
167
  [see them all](https://peter.sh/experiments/chromium-command-line-switches/)
@@ -138,9 +172,9 @@ Ferrum::Browser.new(options)
138
172
  browser process will not be spawned.
139
173
  * `:process_timeout` (Integer) - How long to wait for the Chrome process to
140
174
  respond on startup
141
-
142
-
143
- #### The API below is for master branch and a subject to change before 1.0
175
+ * `:ws_max_receive_size` (Integer) - How big messages to accept from Chrome
176
+ over the web socket, in bytes. Defaults to 64MB. Incoming messages larger
177
+ than this will cause a `Ferrum::DeadBrowserError`.
144
178
 
145
179
 
146
180
  ## Navigation
@@ -186,6 +220,15 @@ browser.goto("https://github.com/")
186
220
  browser.refresh
187
221
  ```
188
222
 
223
+ #### stop
224
+
225
+ Stop all navigations and loading pending resources on the page
226
+
227
+ ```ruby
228
+ browser.goto("https://github.com/")
229
+ browser.stop
230
+ ```
231
+
189
232
 
190
233
  ## Finders
191
234
 
@@ -558,6 +601,8 @@ Sets given values as cookie
558
601
  * :value `String`
559
602
  * :domain `String`
560
603
  * :expires `Integer`
604
+ * :samesite `String`
605
+ * :httponly `Boolean`
561
606
 
562
607
  ```ruby
563
608
  browser.cookies.set(name: "stealth", value: "omg", domain: "google.com") # => true
@@ -686,17 +731,146 @@ browser.evaluate("window.__injected") # => 42
686
731
 
687
732
  ## Frames
688
733
 
689
- #### frames
690
- #### main_frame
691
- #### frame_by
734
+ #### frames : `Array[Frame] | []`
735
+
736
+ Returns all the frames current page have.
737
+
738
+ ```ruby
739
+ browser.goto("https://www.w3schools.com/tags/tag_frame.asp")
740
+ browser.frames # =>
741
+ # [
742
+ # #<Ferrum::Frame @id="C6D104CE454A025FBCF22B98DE612B12" @parent_id=nil @name=nil @state=:stopped_loading @execution_id=1>,
743
+ # #<Ferrum::Frame @id="C09C4E4404314AAEAE85928EAC109A93" @parent_id="C6D104CE454A025FBCF22B98DE612B12" @state=:stopped_loading @execution_id=2>,
744
+ # #<Ferrum::Frame @id="2E9C7F476ED09D87A42F2FEE3C6FBC3C" @parent_id="C6D104CE454A025FBCF22B98DE612B12" @state=:stopped_loading @execution_id=3>,
745
+ # ...
746
+ # ]
747
+ ```
748
+
749
+ #### main_frame : `Frame`
750
+
751
+ Returns page's main frame, the top of the tree and the parent of all frames.
752
+
753
+ #### frame_by(\*\*options) : `Frame | nil`
754
+
755
+ Find frame by given options.
756
+
757
+ * options `Hash`
758
+ * :id `String` - Unique frame's id that browser provides
759
+ * :name `String` - Frame's name if there's one
760
+
761
+ ```ruby
762
+ browser.frame_by(id: "C6D104CE454A025FBCF22B98DE612B12")
763
+ ```
764
+
765
+
766
+ ## Frame
767
+
768
+ #### id : `String`
769
+
770
+ Frame's unique id.
771
+
772
+ #### parent_id : `String | nil`
773
+
774
+ Parent frame id if this one is nested in another one.
775
+
776
+ #### execution_id : `Integer`
777
+
778
+ Execution context id which is used by JS, each frame has it's own context in
779
+ which JS evaluates.
780
+
781
+ #### name : `String | nil`
782
+
783
+ If frame was given a name it should be here.
784
+
785
+ #### state : `Symbol | nil`
786
+
787
+ One of the states frame's in:
788
+
789
+ * `:started_loading`
790
+ * `:navigated`
791
+ * `:stopped_loading`
792
+
793
+ #### url : `String`
794
+
795
+ Returns current frame's location href.
692
796
 
693
- Play around inside given frame
797
+ ```ruby
798
+ browser.goto("https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe")
799
+ frame = browser.frames[1]
800
+ frame.url # => https://interactive-examples.mdn.mozilla.net/pages/tabbed/iframe.html
801
+ ```
802
+
803
+ #### title
804
+
805
+ Returns current frame's title.
806
+
807
+ ```ruby
808
+ browser.goto("https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe")
809
+ frame = browser.frames[1]
810
+ frame.title # => HTML Demo: <iframe>
811
+ ```
812
+
813
+ #### main? : `Boolean`
814
+
815
+ If current frame is the main frame of the page (top of the tree).
816
+
817
+ ```ruby
818
+ browser.goto("https://www.w3schools.com/tags/tag_frame.asp")
819
+ frame = browser.frame_by(id: "C09C4E4404314AAEAE85928EAC109A93")
820
+ frame.main? # => false
821
+ ```
822
+
823
+ #### current_url : `String`
824
+
825
+ Returns current frame's top window location href.
826
+
827
+ ```ruby
828
+ browser.goto("https://www.w3schools.com/tags/tag_frame.asp")
829
+ frame = browser.frame_by(id: "C09C4E4404314AAEAE85928EAC109A93")
830
+ frame.current_url # => "https://www.w3schools.com/tags/tag_frame.asp"
831
+ ```
832
+
833
+ #### current_title : `String`
834
+
835
+ Returns current frame's top window title.
836
+
837
+ ```ruby
838
+ browser.goto("https://www.w3schools.com/tags/tag_frame.asp")
839
+ frame = browser.frame_by(id: "C09C4E4404314AAEAE85928EAC109A93")
840
+ frame.current_title # => "HTML frame tag"
841
+ ```
842
+
843
+ #### body : `String`
844
+
845
+ Returns current frame's html.
846
+
847
+ ```ruby
848
+ browser.goto("https://www.w3schools.com/tags/tag_frame.asp")
849
+ frame = browser.frame_by(id: "C09C4E4404314AAEAE85928EAC109A93")
850
+ frame.body # => "<html><head></head><body></body></html>"
851
+ ```
852
+
853
+ #### doctype
854
+
855
+ Returns current frame's doctype.
856
+
857
+ ```ruby
858
+ browser.goto("https://www.w3schools.com/tags/tag_frame.asp")
859
+ browser.main_frame.doctype # => "<!DOCTYPE html>"
860
+ ```
861
+
862
+ #### set_content(html)
863
+
864
+ Sets a content of a given frame.
865
+
866
+ * html `String`
694
867
 
695
868
  ```ruby
696
869
  browser.goto("https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe")
697
870
  frame = browser.frames[1]
698
- puts frame.title # => HTML Demo: <iframe>
699
- puts frame.url # => https://interactive-examples.mdn.mozilla.net/pages/tabbed/iframe.html
871
+ frame.body # <html lang="en"><head><style>body {transition: opacity ease-in 0.2s; }...
872
+ frame.set_content("<html><head></head><body><p>lol</p></body></html>")
873
+ frame.body # => <html><head></head><body><p>lol</p></body></html>
700
874
  ```
701
875
 
702
876
 
@@ -781,3 +955,27 @@ t2.join
781
955
 
782
956
  browser.quit
783
957
  ```
958
+
959
+
960
+ ## License
961
+
962
+ Copyright 2018-2020 Machinio
963
+
964
+ Permission is hereby granted, free of charge, to any person obtaining
965
+ a copy of this software and associated documentation files (the
966
+ "Software"), to deal in the Software without restriction, including
967
+ without limitation the rights to use, copy, modify, merge, publish,
968
+ distribute, sublicense, and/or sell copies of the Software, and to
969
+ permit persons to whom the Software is furnished to do so, subject to
970
+ the following conditions:
971
+
972
+ The above copyright notice and this permission notice shall be
973
+ included in all copies or substantial portions of the Software.
974
+
975
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
976
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
977
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
978
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
979
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
980
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
981
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -36,6 +36,12 @@ module Ferrum
36
36
  end
37
37
  end
38
38
 
39
+ class ProcessTimeoutError < Error
40
+ def initialize(timeout)
41
+ super("Browser did not produce websocket url within #{timeout} seconds")
42
+ end
43
+ end
44
+
39
45
  class DeadBrowserError < Error
40
46
  def initialize(message = "Browser is dead or given window is closed")
41
47
  super
@@ -72,8 +78,8 @@ module Ferrum
72
78
  attr_reader :class_name, :message
73
79
 
74
80
  def initialize(response)
75
- super
76
81
  @class_name, @message = response.values_at("className", "description")
82
+ super(response.merge("message" => @message))
77
83
  end
78
84
  end
79
85
 
@@ -16,8 +16,9 @@ module Ferrum
16
16
  extend Forwardable
17
17
  delegate %i[default_context] => :contexts
18
18
  delegate %i[targets create_target create_page page pages windows] => :default_context
19
- delegate %i[goto back forward refresh reload
20
- at_css at_xpath css xpath current_url title body doctype
19
+ delegate %i[goto back forward refresh reload stop
20
+ at_css at_xpath css xpath current_url current_title url title
21
+ body doctype set_content
21
22
  headers cookies network
22
23
  mouse keyboard
23
24
  screenshot pdf viewport_size
@@ -28,7 +29,7 @@ module Ferrum
28
29
  delegate %i[default_user_agent] => :process
29
30
 
30
31
  attr_reader :client, :process, :contexts, :logger, :js_errors,
31
- :slowmo, :base_url, :options, :window_size
32
+ :slowmo, :base_url, :options, :window_size, :ws_max_receive_size
32
33
  attr_writer :timeout
33
34
 
34
35
  def initialize(options = nil)
@@ -39,9 +40,10 @@ module Ferrum
39
40
  @original_window_size = @window_size
40
41
 
41
42
  @options = Hash(options.merge(window_size: @window_size))
42
- @logger, @timeout = @options.values_at(:logger, :timeout)
43
+ @logger, @timeout, @ws_max_receive_size =
44
+ @options.values_at(:logger, :timeout, :ws_max_receive_size)
43
45
  @js_errors = @options.fetch(:js_errors, false)
44
- @slowmo = @options[:slowmo].to_i
46
+ @slowmo = @options[:slowmo].to_f
45
47
 
46
48
  if @options.key?(:base_url)
47
49
  self.base_url = @options[:base_url]
@@ -114,7 +116,7 @@ module Ferrum
114
116
  def start
115
117
  Ferrum.started
116
118
  @process = Process.start(@options)
117
- @client = Client.new(self, @process.ws_url, 0, false)
119
+ @client = Client.new(self, @process.ws_url)
118
120
  @contexts = Contexts.new(self)
119
121
  end
120
122
  end
@@ -9,12 +9,11 @@ module Ferrum
9
9
  class Client
10
10
  INTERRUPTIONS = %w[Fetch.requestPaused Fetch.authRequired].freeze
11
11
 
12
- def initialize(browser, ws_url, start_id = 0, allow_slowmo = true)
13
- @command_id = start_id
14
- @pendings = Concurrent::Hash.new
12
+ def initialize(browser, ws_url, id_starts_with: 0)
15
13
  @browser = browser
16
- @slowmo = @browser.slowmo if allow_slowmo && @browser.slowmo > 0
17
- @ws = WebSocket.new(ws_url, @browser.logger)
14
+ @command_id = id_starts_with
15
+ @pendings = Concurrent::Hash.new
16
+ @ws = WebSocket.new(ws_url, @browser.ws_max_receive_size, @browser.logger)
18
17
  @subscriber, @interruptor = Subscriber.build(2)
19
18
 
20
19
  @thread = Thread.new do
@@ -39,7 +38,6 @@ module Ferrum
39
38
  pending = Concurrent::IVar.new
40
39
  message = build_message(method, params)
41
40
  @pendings[message[:id]] = pending
42
- sleep(@slowmo) if @slowmo
43
41
  @ws.send_message(message)
44
42
  data = pending.value!(@browser.timeout)
45
43
  @pendings.delete(message[:id])
@@ -133,8 +133,8 @@ module Ferrum
133
133
  end
134
134
 
135
135
  unless ws_url
136
- @logger.puts output if @logger
137
- raise "Browser process did not produce websocket url within #{timeout} seconds"
136
+ @logger.puts(output) if @logger
137
+ raise ProcessTimeoutError.new(timeout)
138
138
  end
139
139
  end
140
140
 
@@ -11,12 +11,13 @@ module Ferrum
11
11
 
12
12
  attr_reader :url, :messages
13
13
 
14
- def initialize(url, logger)
14
+ def initialize(url, max_receive_size, logger)
15
15
  @url = url
16
16
  @logger = logger
17
17
  uri = URI.parse(@url)
18
18
  @sock = TCPSocket.new(uri.host, uri.port)
19
- @driver = ::WebSocket::Driver.client(self)
19
+ max_receive_size ||= ::WebSocket::Driver::MAX_LENGTH
20
+ @driver = ::WebSocket::Driver.client(self, max_length: max_receive_size)
20
21
  @messages = Queue.new
21
22
 
22
23
  @driver.on(:open, &method(:on_open))
@@ -25,7 +26,9 @@ module Ferrum
25
26
 
26
27
  @thread = Thread.new do
27
28
  Thread.current.abort_on_exception = true
28
- Thread.current.report_on_exception = true if Thread.current.respond_to?(:report_on_exception=)
29
+ if Thread.current.respond_to?(:report_on_exception=)
30
+ Thread.current.report_on_exception = true
31
+ end
29
32
 
30
33
  begin
31
34
  while data = @sock.readpartial(512)
@@ -23,6 +23,10 @@ module Ferrum
23
23
  @attributes["path"]
24
24
  end
25
25
 
26
+ def samesite
27
+ @attributes["sameSite"]
28
+ end
29
+
26
30
  def size
27
31
  @attributes["size"]
28
32
  end
@@ -65,6 +69,9 @@ module Ferrum
65
69
  cookie[:value] ||= value
66
70
  cookie[:domain] ||= default_domain
67
71
 
72
+ cookie[:httpOnly] = cookie.delete(:httponly) if cookie.key?(:httponly)
73
+ cookie[:sameSite] = cookie.delete(:samesite) if cookie.key?(:samesite)
74
+
68
75
  expires = cookie.delete(:expires).to_i
69
76
  cookie[:expires] = expires if expires > 0
70
77
 
@@ -14,11 +14,11 @@ module Ferrum
14
14
  options = { accept: true }
15
15
  response = prompt_text || default_prompt
16
16
  options.merge!(promptText: response) if response
17
- @page.command("Page.handleJavaScriptDialog", **options)
17
+ @page.command("Page.handleJavaScriptDialog", slowmoable: true, **options)
18
18
  end
19
19
 
20
20
  def dismiss
21
- @page.command("Page.handleJavaScriptDialog", accept: false)
21
+ @page.command("Page.handleJavaScriptDialog", slowmoable: true, accept: false)
22
22
  end
23
23
 
24
24
  def match?(regexp)
@@ -7,9 +7,8 @@ module Ferrum
7
7
  class Frame
8
8
  include DOM, Runtime
9
9
 
10
- attr_reader :id, :page, :parent_id, :state
11
- attr_writer :execution_id
12
- attr_accessor :name
10
+ attr_reader :page, :parent_id, :state
11
+ attr_accessor :id, :name
13
12
 
14
13
  def initialize(id, page, parent_id = nil)
15
14
  @id, @page, @parent_id = id, page, parent_id
@@ -35,6 +34,15 @@ module Ferrum
35
34
  @parent_id.nil?
36
35
  end
37
36
 
37
+ def set_content(html)
38
+ evaluate_async(%(
39
+ document.open();
40
+ document.write(arguments[0]);
41
+ document.close();
42
+ arguments[1](true);
43
+ ), @page.timeout, html)
44
+ end
45
+
38
46
  def execution_id?(execution_id)
39
47
  @execution_id == execution_id
40
48
  end
@@ -47,6 +55,14 @@ module Ferrum
47
55
  @page.event.wait(@page.timeout) ? retry : raise
48
56
  end
49
57
 
58
+ def set_execution_id(value)
59
+ @execution_id ||= value
60
+ end
61
+
62
+ def reset_execution_id
63
+ @execution_id = nil
64
+ end
65
+
50
66
  def inspect
51
67
  %(#<#{self.class} @id=#{@id.inspect} @parent_id=#{@parent_id.inspect} @name=#{@name.inspect} @state=#{@state.inspect} @execution_id=#{@execution_id.inspect}>)
52
68
  end
@@ -29,65 +29,62 @@ module Ferrum
29
29
  end
30
30
 
31
31
  def doctype
32
- evaluate("new XMLSerializer().serializeToString(document.doctype)")
32
+ evaluate("document.doctype && new XMLSerializer().serializeToString(document.doctype)")
33
33
  end
34
34
 
35
35
  def body
36
36
  evaluate("document.documentElement.outerHTML")
37
37
  end
38
38
 
39
- def at_xpath(selector, within: nil)
40
- xpath(selector, within: within).first
41
- end
42
-
43
- # FIXME: Check within
44
39
  def xpath(selector, within: nil)
45
- evaluate_async(%(
46
- try {
47
- let selector = arguments[0];
48
- let within = arguments[1] || document;
49
- let results = [];
40
+ code = <<~JS
41
+ let selector = arguments[0];
42
+ let within = arguments[1] || document;
43
+ let results = [];
50
44
 
51
- let xpath = document.evaluate(selector, within, null, XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null);
52
- for (let i = 0; i < xpath.snapshotLength; i++) {
53
- results.push(xpath.snapshotItem(i));
54
- }
45
+ let xpath = document.evaluate(selector, within, null, XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null);
46
+ for (let i = 0; i < xpath.snapshotLength; i++) {
47
+ results.push(xpath.snapshotItem(i));
48
+ }
55
49
 
56
- arguments[2](results);
57
- } catch (error) {
58
- // DOMException.INVALID_EXPRESSION_ERR is undefined, using pure code
59
- if (error.code == DOMException.SYNTAX_ERR || error.code == 51) {
60
- throw "Invalid Selector";
61
- } else {
62
- throw error;
63
- }
64
- }), @page.timeout, selector, within)
50
+ arguments[2](results);
51
+ JS
52
+
53
+ evaluate_async(code, @page.timeout, selector, within)
65
54
  end
66
55
 
67
- # FIXME css doesn't work for a frame w/o execution_id
68
- def css(selector, within: nil)
69
- node_id = within&.node_id || @page.document_id
56
+ def at_xpath(selector, within: nil)
57
+ code = <<~JS
58
+ let selector = arguments[0];
59
+ let within = arguments[1] || document;
60
+ let xpath = document.evaluate(selector, within, null, XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null);
61
+ let result = xpath.snapshotItem(0);
62
+ arguments[2](result);
63
+ JS
70
64
 
71
- ids = @page.command("DOM.querySelectorAll",
72
- nodeId: node_id,
73
- selector: selector)["nodeIds"]
74
- ids.map { |id| build_node(id) }.compact
65
+ evaluate_async(code, @page.timeout, selector, within)
75
66
  end
76
67
 
77
- def at_css(selector, within: nil)
78
- node_id = within&.node_id || @page.document_id
68
+ def css(selector, within: nil)
69
+ code = <<~JS
70
+ let selector = arguments[0];
71
+ let within = arguments[1] || document;
72
+ let results = within.querySelectorAll(selector);
73
+ arguments[2](results);
74
+ JS
79
75
 
80
- id = @page.command("DOM.querySelector",
81
- nodeId: node_id,
82
- selector: selector)["nodeId"]
83
- build_node(id)
76
+ evaluate_async(code, @page.timeout, selector, within)
84
77
  end
85
78
 
86
- private
79
+ def at_css(selector, within: nil)
80
+ code = <<~JS
81
+ let selector = arguments[0];
82
+ let within = arguments[1] || document;
83
+ let result = within.querySelector(selector);
84
+ arguments[2](result);
85
+ JS
87
86
 
88
- def build_node(node_id)
89
- description = @page.command("DOM.describeNode", nodeId: node_id)
90
- Node.new(self, @page.target_id, node_id, description["node"])
87
+ evaluate_async(code, @page.timeout, selector, within)
91
88
  end
92
89
  end
93
90
  end
@@ -85,8 +85,10 @@ module Ferrum
85
85
  options.merge!(returnByValue: by_value)
86
86
 
87
87
  response = @page.command("Runtime.callFunctionOn",
88
- wait: wait, **options)["result"]
89
- .tap { |r| handle_error(r) }
88
+ wait: wait, slowmoable: true,
89
+ **options)
90
+ handle_error(response)
91
+ response = response["result"]
90
92
 
91
93
  by_value ? response.dig("value") : handle_response(response)
92
94
  end
@@ -137,14 +139,18 @@ module Ferrum
137
139
  end
138
140
 
139
141
  response = @page.command("Runtime.callFunctionOn",
140
- **params)["result"].tap { |r| handle_error(r) }
142
+ slowmoable: true,
143
+ **params)
144
+ handle_error(response)
145
+ response = response["result"]
141
146
 
142
147
  handle ? handle_response(response) : response
143
148
  end
144
149
  end
145
150
 
146
151
  # FIXME: We should have a central place to handle all type of errors
147
- def handle_error(result)
152
+ def handle_error(response)
153
+ result = response["result"]
148
154
  return if result["subtype"] != "error"
149
155
 
150
156
  case result["description"]
@@ -220,37 +226,38 @@ module Ferrum
220
226
  end
221
227
 
222
228
  def cyclic?(object_id)
223
- @page.command("Runtime.callFunctionOn",
224
- objectId: object_id,
225
- returnByValue: true,
226
- functionDeclaration: <<~JS
227
- function() {
228
- if (Array.isArray(this) &&
229
- this.every(e => e instanceof Node)) {
230
- return false;
231
- }
232
-
233
- const seen = [];
234
- function detectCycle(obj) {
235
- if (typeof obj === "object") {
236
- if (seen.indexOf(obj) !== -1) {
237
- return true;
238
- }
239
- seen.push(obj);
240
- for (let key in obj) {
241
- if (obj.hasOwnProperty(key) && detectCycle(obj[key])) {
242
- return true;
243
- }
244
- }
245
- }
229
+ @page.command(
230
+ "Runtime.callFunctionOn",
231
+ objectId: object_id,
232
+ returnByValue: true,
233
+ functionDeclaration: <<~JS
234
+ function() {
235
+ if (Array.isArray(this) &&
236
+ this.every(e => e instanceof Node)) {
237
+ return false;
238
+ }
246
239
 
247
- return false;
240
+ const seen = [];
241
+ function detectCycle(obj) {
242
+ if (typeof obj === "object") {
243
+ if (seen.indexOf(obj) !== -1) {
244
+ return true;
245
+ }
246
+ seen.push(obj);
247
+ for (let key in obj) {
248
+ if (obj.hasOwnProperty(key) && detectCycle(obj[key])) {
249
+ return true;
248
250
  }
249
-
250
- return detectCycle(this);
251
251
  }
252
- JS
253
- )
252
+ }
253
+
254
+ return false;
255
+ }
256
+
257
+ return detectCycle(this);
258
+ }
259
+ JS
260
+ )
254
261
  end
255
262
  end
256
263
  end
@@ -33,13 +33,13 @@ module Ferrum
33
33
  def down(key)
34
34
  key = normalize_keys(Array(key))
35
35
  type = key[:text] ? "keyDown" : "rawKeyDown"
36
- @page.command("Input.dispatchKeyEvent", type: type, **key)
36
+ @page.command("Input.dispatchKeyEvent", slowmoable: true, type: type, **key)
37
37
  self
38
38
  end
39
39
 
40
40
  def up(key)
41
41
  key = normalize_keys(Array(key))
42
- @page.command("Input.dispatchKeyEvent", type: "keyUp", **key)
42
+ @page.command("Input.dispatchKeyEvent", slowmoable: true, type: "keyUp", **key)
43
43
  self
44
44
  end
45
45
 
@@ -49,7 +49,7 @@ module Ferrum
49
49
  keys.each do |key|
50
50
  type = key[:text] ? "keyDown" : "rawKeyDown"
51
51
  @page.command("Input.dispatchKeyEvent", type: type, **key)
52
- @page.command("Input.dispatchKeyEvent", type: "keyUp", **key)
52
+ @page.command("Input.dispatchKeyEvent", slowmoable: true, type: "keyUp", **key)
53
53
  end
54
54
 
55
55
  self
@@ -32,10 +32,21 @@ module Ferrum
32
32
  tap { mouse_event(type: "mouseReleased", **options) }
33
33
  end
34
34
 
35
- # FIXME: steps
36
35
  def move(x:, y:, steps: 1)
36
+ from_x, from_y = @x, @y
37
37
  @x, @y = x, y
38
- @page.command("Input.dispatchMouseEvent", type: "mouseMoved", x: @x, y: @y)
38
+
39
+ steps.times do |i|
40
+ new_x = from_x + (@x - from_x) * ((i + 1) / steps.to_f)
41
+ new_y = from_y + (@y - from_y) * ((i + 1) / steps.to_f)
42
+
43
+ @page.command("Input.dispatchMouseEvent",
44
+ slowmoable: true,
45
+ type: "mouseMoved",
46
+ x: new_x.to_i,
47
+ y: new_y.to_i)
48
+ end
49
+
39
50
  self
40
51
  end
41
52
 
@@ -45,7 +56,7 @@ module Ferrum
45
56
  button = validate_button(button)
46
57
  options = { x: @x, y: @y, type: type, button: button, clickCount: count }
47
58
  options.merge!(modifiers: modifiers) if modifiers
48
- @page.command("Input.dispatchMouseEvent", wait: wait, **options)
59
+ @page.command("Input.dispatchMouseEvent", wait: wait, slowmoable: true, **options)
49
60
  end
50
61
 
51
62
  def validate_button(button)
@@ -24,7 +24,7 @@ module Ferrum
24
24
  end
25
25
 
26
26
  def focus
27
- tap { page.command("DOM.focus", nodeId: node_id) }
27
+ tap { page.command("DOM.focus", slowmoable: true, nodeId: node_id) }
28
28
  end
29
29
 
30
30
  def blur
@@ -37,22 +37,23 @@ module Ferrum
37
37
 
38
38
  # mode: (:left | :right | :double)
39
39
  # keys: (:alt, (:ctrl | :control), (:meta | :command), :shift)
40
- # offset: { :x, :y }
41
- def click(mode: :left, keys: [], offset: {})
42
- x, y = find_position(offset[:x], offset[:y])
40
+ # offset: { :x, :y, :position (:top | :center) }
41
+ def click(mode: :left, keys: [], offset: {}, delay: 0)
42
+ x, y = find_position(**offset)
43
43
  modifiers = page.keyboard.modifiers(keys)
44
44
 
45
45
  case mode
46
46
  when :right
47
47
  page.mouse.move(x: x, y: y)
48
48
  page.mouse.down(button: :right, modifiers: modifiers)
49
+ sleep(delay)
49
50
  page.mouse.up(button: :right, modifiers: modifiers)
50
51
  when :double
51
52
  page.mouse.move(x: x, y: y)
52
53
  page.mouse.down(modifiers: modifiers, count: 2)
53
54
  page.mouse.up(modifiers: modifiers, count: 2)
54
55
  when :left
55
- page.mouse.click(x: x, y: y, modifiers: modifiers)
56
+ page.mouse.click(x: x, y: y, modifiers: modifiers, delay: delay)
56
57
  end
57
58
 
58
59
  self
@@ -63,7 +64,7 @@ module Ferrum
63
64
  end
64
65
 
65
66
  def select_file(value)
66
- page.command("DOM.setFileInputFiles", nodeId: node_id, files: Array(value))
67
+ page.command("DOM.setFileInputFiles", slowmoable: true, nodeId: node_id, files: Array(value))
67
68
  end
68
69
 
69
70
  def at_xpath(selector)
@@ -119,20 +120,31 @@ module Ferrum
119
120
  %(#<#{self.class} @target_id=#{@target_id.inspect} @node_id=#{@node_id} @description=#{@description.inspect}>)
120
121
  end
121
122
 
122
- def find_position(offset_x = nil, offset_y = nil)
123
+ def find_position(x: nil, y: nil, position: :top)
124
+ offset_x, offset_y = x, y
123
125
  quads = get_content_quads
124
- offset_x, offset_y = offset_x.to_i, offset_y.to_i
126
+ x = y = nil
125
127
 
126
- if offset_x > 0 || offset_y > 0
128
+ if offset_x && offset_y && position == :top
127
129
  point = quads.first
128
- [point[:x] + offset_x, point[:y] + offset_y]
130
+ x = point[:x] + offset_x.to_i
131
+ y = point[:y] + offset_y.to_i
129
132
  else
130
133
  x, y = quads.inject([0, 0]) do |memo, point|
131
134
  [memo[0] + point[:x],
132
135
  memo[1] + point[:y]]
133
136
  end
134
- [x / 4, y / 4]
137
+
138
+ x = x / 4
139
+ y = y / 4
140
+ end
141
+
142
+ if offset_x && offset_y && position == :center
143
+ x = x + offset_x.to_i
144
+ y = y + offset_y.to_i
135
145
  end
146
+
147
+ [x, y]
136
148
  end
137
149
 
138
150
  private
@@ -31,7 +31,7 @@ module Ferrum
31
31
 
32
32
  extend Forwardable
33
33
  delegate %i[at_css at_xpath css xpath
34
- current_url current_title url title body doctype
34
+ current_url current_title url title body doctype set_content
35
35
  execution_id evaluate evaluate_on evaluate_async execute
36
36
  add_script_tag add_style_tag] => :main_frame
37
37
 
@@ -44,13 +44,14 @@ module Ferrum
44
44
 
45
45
  def initialize(target_id, browser)
46
46
  @frames = {}
47
+ @main_frame = Frame.new(nil, self)
47
48
  @target_id, @browser = target_id, browser
48
49
  @event = Event.new.tap(&:set)
49
50
 
50
51
  host = @browser.process.host
51
52
  port = @browser.process.port
52
53
  ws_url = "ws://#{host}:#{port}/devtools/page/#{@target_id}"
53
- @client = Browser::Client.new(browser, ws_url, 1000)
54
+ @client = Browser::Client.new(browser, ws_url, id_starts_with: 1000)
54
55
 
55
56
  @mouse, @keyboard = Mouse.new(self), Keyboard.new(self)
56
57
  @headers, @cookies = Headers.new(self), Cookies.new(self)
@@ -99,7 +100,8 @@ module Ferrum
99
100
  @browser.command("Browser.setWindowBounds", windowId: @window_id, bounds: { width: width, height: height, windowState: "normal" })
100
101
  end
101
102
 
102
- command("Emulation.setDeviceMetricsOverride", width: width,
103
+ command("Emulation.setDeviceMetricsOverride", slowmoable: true,
104
+ width: width,
103
105
  height: height,
104
106
  deviceScaleFactor: 1,
105
107
  mobile: false,
@@ -107,10 +109,14 @@ module Ferrum
107
109
  end
108
110
 
109
111
  def refresh
110
- command("Page.reload", wait: timeout)
112
+ command("Page.reload", wait: timeout, slowmoable: true)
111
113
  end
112
114
  alias_method :reload, :refresh
113
115
 
116
+ def stop
117
+ command("Page.stopLoading", slowmoable: true)
118
+ end
119
+
114
120
  def back
115
121
  history_navigate(delta: -1)
116
122
  end
@@ -125,9 +131,11 @@ module Ferrum
125
131
  enabled
126
132
  end
127
133
 
128
- def command(method, wait: 0, **params)
134
+ def command(method, wait: 0, slowmoable: false, **params)
129
135
  iteration = @event.reset if wait > 0
136
+ sleep(@browser.slowmo) if slowmoable && @browser.slowmo > 0
130
137
  result = @client.command(method, params)
138
+
131
139
  if wait > 0
132
140
  @event.wait(wait) # Wait a bit after command and check if iteration has
133
141
  # changed which means there was some network event for
@@ -236,7 +244,9 @@ module Ferrum
236
244
 
237
245
  if entry = entries[index + delta]
238
246
  # Potential wait because of network event
239
- command("Page.navigateToHistoryEntry", wait: Mouse::CLICK_WAIT, entryId: entry["id"])
247
+ command("Page.navigateToHistoryEntry", wait: Mouse::CLICK_WAIT,
248
+ slowmoable: true,
249
+ entryId: entry["id"])
240
250
  end
241
251
  end
242
252
 
@@ -73,21 +73,27 @@ module Ferrum
73
73
  on("Runtime.executionContextCreated") do |params|
74
74
  context_id = params.dig("context", "id")
75
75
  frame_id = params.dig("context", "auxData", "frameId")
76
+
77
+ unless @main_frame.id
78
+ @main_frame.id = frame_id
79
+ @frames[frame_id] = @main_frame
80
+ end
81
+
76
82
  frame = @frames[frame_id] || Frame.new(frame_id, self)
77
- frame.execution_id = context_id
83
+ frame.set_execution_id(context_id)
78
84
 
79
- @main_frame ||= frame
80
85
  @frames[frame_id] ||= frame
81
86
  end
82
87
 
83
88
  on("Runtime.executionContextDestroyed") do |params|
84
89
  execution_id = params["executionContextId"]
85
90
  frame = frames.find { |f| f.execution_id?(execution_id) }
86
- frame.execution_id = nil
91
+ frame.reset_execution_id
87
92
  end
88
93
 
89
94
  on("Runtime.executionContextsCleared") do
90
95
  @frames.delete_if { |_, f| !f.main? }
96
+ @main_frame.reset_execution_id
91
97
  end
92
98
  end
93
99
 
@@ -48,8 +48,8 @@ module Ferrum
48
48
 
49
49
  def document_size
50
50
  evaluate <<~JS
51
- [document.documentElement.offsetWidth,
52
- document.documentElement.offsetHeight]
51
+ [document.documentElement.scrollWidth,
52
+ document.documentElement.scrollHeight]
53
53
  JS
54
54
  end
55
55
 
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Ferrum
4
- VERSION = "0.7"
4
+ VERSION = "0.8"
5
5
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: ferrum
3
3
  version: !ruby/object:Gem::Version
4
- version: '0.7'
4
+ version: '0.8'
5
5
  platform: ruby
6
6
  authors:
7
7
  - Dmitry Vorotilin
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2020-01-28 00:00:00.000000000 Z
11
+ date: 2020-04-07 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: websocket-driver
@@ -231,7 +231,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
231
231
  - !ruby/object:Gem::Version
232
232
  version: '0'
233
233
  requirements: []
234
- rubygems_version: 3.0.3
234
+ rubygems_version: 3.1.2
235
235
  signing_key:
236
236
  specification_version: 4
237
237
  summary: Ruby headless Chrome driver