intransient_capybara 1.0.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/.gitignore +10 -0
- data/Gemfile +4 -0
- data/README.md +131 -0
- data/Rakefile +12 -0
- data/bin/console +14 -0
- data/bin/setup +7 -0
- data/intransient_capybara.gemspec +31 -0
- data/lib/intransient_capybara.rb +9 -0
- data/lib/intransient_capybara/intransient_capybara_helper.rb +105 -0
- data/lib/intransient_capybara/intransient_capybara_test.rb +121 -0
- data/lib/intransient_capybara/minitest_retry.rb +54 -0
- data/lib/intransient_capybara/patches.rb +61 -0
- data/lib/intransient_capybara/rack_request_blocker.rb +57 -0
- data/lib/intransient_capybara/version.rb +3 -0
- metadata +159 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: ecdd544f27a9c880faed98ed328175998d8e11c7
|
4
|
+
data.tar.gz: d6dca5ca326e3bc8270f71be864584a019cef776
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: c63ae6d269c9f80ac0c4b18efe147603d26bef928b2bdf970a72a2d93ac284316fa2a9ef1111cae083c13c570cb6a9fdd943a1edd5c96e63085247adbe002dfa
|
7
|
+
data.tar.gz: fc101584686f8a6c700cdff4fa1104bfe7568fc73d06251be6df21b829994a474a80b14abbec9526653028967b9fd8d2dbdcce997ad1ba74d9bc7740cad83ae1
|
data/.gitignore
ADDED
data/Gemfile
ADDED
data/README.md
ADDED
@@ -0,0 +1,131 @@
|
|
1
|
+
# Intransient Capybara
|
2
|
+
|
3
|
+
This gem provides a set of improvements to the commonly used test technology stack of Minitest, Capybara, Poltergeist, and PhantomJS for rails applications, intended to reduce the occurrence of transient failures.
|
4
|
+
|
5
|
+
This gem is inspired by all of these posts:
|
6
|
+
|
7
|
+
https://bibwild.wordpress.com/2016/02/18/struggling-towards-reliable-capybara-javascript-testing/
|
8
|
+
https://semaphoreci.com/community/tutorials/how-to-deal-with-and-eliminate-flaky-tests
|
9
|
+
http://www.urbanbound.com/make/fix-flaky-feature-tests-by-using-capybaras-apis-properly
|
10
|
+
http://johnpwood.net/2015/04/23/tips-and-tricks-for-dubugging-and-fixing-slowflaky-capybara-specs/
|
11
|
+
http://tech.simplybusiness.co.uk/2015/02/25/flaky-tests-and-capybara-best-practices/
|
12
|
+
https://www.joinhandshake.com/engineering/2016/03/15/tackling-flaky-capybara-tests.html
|
13
|
+
https://robots.thoughtbot.com/write-reliable-asynchronous-integration-tests-with-capybara
|
14
|
+
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/2646/diffs
|
15
|
+
https://github.com/appfolio/minitest-optional_retry
|
16
|
+
|
17
|
+
## Installation
|
18
|
+
|
19
|
+
Add this line to your application's Gemfile:
|
20
|
+
|
21
|
+
```ruby
|
22
|
+
gem 'intransient_capybara'
|
23
|
+
```
|
24
|
+
|
25
|
+
And then execute:
|
26
|
+
|
27
|
+
$ bundle
|
28
|
+
|
29
|
+
Or install it yourself as:
|
30
|
+
|
31
|
+
$ gem install intransient_capybara
|
32
|
+
|
33
|
+
## Usage
|
34
|
+
|
35
|
+
Inherit from IntransientCapybaraTest in all your integration tests
|
36
|
+
|
37
|
+
```ruby
|
38
|
+
require 'intransient_capybara/intransient_capybara_test'
|
39
|
+
|
40
|
+
class MyAppsIntegrationTestClass < IntransientCapybaraTest
|
41
|
+
|
42
|
+
def setup
|
43
|
+
set_up_mocks
|
44
|
+
|
45
|
+
# This ought to be called last
|
46
|
+
super
|
47
|
+
end
|
48
|
+
|
49
|
+
def teardown
|
50
|
+
# This *really* should be called first because it stops requests before you clear stuff out
|
51
|
+
super
|
52
|
+
|
53
|
+
clear_out_cache
|
54
|
+
clear_out_redis
|
55
|
+
clear_out_class_variables
|
56
|
+
clear_out_job_queue
|
57
|
+
DatabaseCleaner.clean
|
58
|
+
end
|
59
|
+
|
60
|
+
end
|
61
|
+
```
|
62
|
+
|
63
|
+
Include RackRequestBlocker, in application.rb.
|
64
|
+
|
65
|
+
```ruby
|
66
|
+
if Rails.env.test?
|
67
|
+
require 'intransient_capybara/rack_request_blocker'
|
68
|
+
config.middleware.insert_before('ActionDispatch::Static', 'RackRequestBlocker')
|
69
|
+
end
|
70
|
+
```
|
71
|
+
|
72
|
+
Register the driver
|
73
|
+
|
74
|
+
```ruby
|
75
|
+
|
76
|
+
Capybara.register_driver :poltergeist do |app|
|
77
|
+
Capybara::Poltergeist::Driver.new(app,
|
78
|
+
{
|
79
|
+
phantomjs_logger: StringIO.new,
|
80
|
+
logger: StringIO.new,
|
81
|
+
timeout: 60,
|
82
|
+
debug: ENV['DEBUG_POLTERGEIST'],
|
83
|
+
phantomjs_options:
|
84
|
+
[
|
85
|
+
'--load-images=no',
|
86
|
+
'--ignore-ssl-errors=yes',
|
87
|
+
],
|
88
|
+
}
|
89
|
+
)
|
90
|
+
end
|
91
|
+
|
92
|
+
```
|
93
|
+
|
94
|
+
###Debuggability
|
95
|
+
|
96
|
+
1. When an element is not found, we output what the current path is and the page's entire HTML, to debug what is going on. A common case is getting redirected somewhere else, and another is the wrong template is rendering or your partial is not rendering at all. Sometimes you look at the HTML and your element is there and knowing that will help figure out why the selector isn't working.
|
97
|
+
|
98
|
+
2. When environment variable DEBUG_TEST_TRAFFIC is 'true', we output a summary of network traffic from each test. If you are downloading a lot of stuff or making inordinant amounts of calls it will be insightful to you to know this - it might be reducable, or you may need to preload stuff, etc. You can also set DEBUG_TEST_TRAFFIC_RAISE_EXTERNAL to 'true' and we will throw an exception if you make a network call to an external service. This is super common and you should either stub out all of these or add them to the blacklist. jQuery is a common one if you are loading that JS from a CDN, and you may need to load it from the asset pipeline during tests instead.
|
99
|
+
|
100
|
+
3. When environment variable TRACE_TEST_FRAMEWORK is 'true', we output some traces of blocking rack requests and the calling of test setup and teardown. This can help you make sure your ordering is correct and could help you see if things are happening concurrently or not.
|
101
|
+
|
102
|
+
###Correct configuration
|
103
|
+
|
104
|
+
1. Capybara.default_max_wait_time is a very important decision. This gem defaults that to 10 seconds. More than that and you're probably failing anyways and you're just extending your test run for no reason. Much less and you get transient. Time is less important than stability and 10 seconds is a good balance on the stabiltiy side of the world. You can override this by setting Capybara.default_max_wait_time again in your inherited class.
|
105
|
+
|
106
|
+
2. Set phantomjs_options when you set up Poltergeist. Not loading images and ignoring SSL errors are the common suggestions to improve stability in PhantomJS.
|
107
|
+
|
108
|
+
###Correct usage of Capybara/Poltergeist/PhantomJS
|
109
|
+
|
110
|
+
1. You need to setup and teardown consistently in your tests, so we will throw an exception if you override one of these and forget to call super (which is SUPER common).
|
111
|
+
|
112
|
+
2. When you reach a timeout, a warning is printed. Sometimes writing a test wrong will always reach the timeout so this helps to improve test performance. This is similar to capybara-slow_finder_errors but since we are patching this method for another reason as well we also patch this part. https://github.com/ngauthier/capybara-slow_finder_errors.
|
113
|
+
|
114
|
+
3. A lot of stuff gets downloaded on the first page load, so a lot of transient tests are because of the very first page load. We will load a page for you before we run tests to fix this. The root path is default, and you can override it with IntransientCapybaraTest.cache_warmup_path = 'my path'.
|
115
|
+
|
116
|
+
### Capybara/Poltergeist/PhantomJS improvements
|
117
|
+
|
118
|
+
1. Rack request blocker - http://blog.salsify.com/engineering/tearing-capybara-ajax-tests. TLDR, use middleware to trace server calls so we KNOW when requests are complete before moving on to the next test. This helps a LOT. For example, if you clear out mocks in your teardown method, and you call teardown while an AJAX request is still running, you can have problems from missing mocks because of this race condition.
|
119
|
+
|
120
|
+
2. Minitest retry. This is two parts. First, no matter what you do, sometimes PhantomJS dies on you. We catch DeadClient and will retry a test once if it gets thrown. Next, more controversially, even after all the improvements represented here and elsewhere, sometimes we get transient tests. We do our best, but they're there. Most of the time we re-run them and move on. Why waste time re-running them manually? We will run tests the number of times the environment variable MINITEST_RETRY_COUNT is set to, defaulting to 3 times (so, 2 retries for every test by default). If it fails at first then succeeds, we will output that it is a transient test and report the success of the test. You can deactivate this by setting the environment variable TRANSIENT_TESTS_REPORT_FAILURE to 'true'. This way you could still see that tests are transient because the re-runs passed and continue to fail transient tests if you so prefer.
|
121
|
+
|
122
|
+
## Development
|
123
|
+
|
124
|
+
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake false` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
|
125
|
+
|
126
|
+
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
|
127
|
+
|
128
|
+
## Contributing
|
129
|
+
|
130
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/avvo/intransient_capybara.
|
131
|
+
|
data/Rakefile
ADDED
data/bin/console
ADDED
@@ -0,0 +1,14 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
|
3
|
+
require "bundler/setup"
|
4
|
+
require "easygoing_active_model"
|
5
|
+
|
6
|
+
# You can add fixtures and/or initialization code here to make experimenting
|
7
|
+
# with your gem easier. You can also use a different console, if you like.
|
8
|
+
|
9
|
+
# (If you use this, don't forget to add pry to your Gemfile!)
|
10
|
+
# require "pry"
|
11
|
+
# Pry.start
|
12
|
+
|
13
|
+
require "irb"
|
14
|
+
IRB.start
|
data/bin/setup
ADDED
@@ -0,0 +1,31 @@
|
|
1
|
+
# coding: utf-8
|
2
|
+
lib = File.expand_path('../lib', __FILE__)
|
3
|
+
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
4
|
+
|
5
|
+
require 'intransient_capybara/version'
|
6
|
+
|
7
|
+
Gem::Specification.new do |spec|
|
8
|
+
spec.name = "intransient_capybara"
|
9
|
+
spec.version = IntransientCapybara::VERSION
|
10
|
+
spec.authors = ["Seth Ringling"]
|
11
|
+
spec.email = ["sringling@avvo.com"]
|
12
|
+
|
13
|
+
spec.summary = %q{A set of improvements to Capybara/Poltergeist/PhantomJS test stack that reduces the occurrence transient failures.}
|
14
|
+
spec.description = %q{With improved debuggability, with proper usage and configuration of Capybara/Poltergeist/PhantomJS, and with some improvements on top of it we can greatly reduce the occurrence of transient integration/UI test failures.}
|
15
|
+
spec.homepage = "https://github.com/avvo/intransient_capybara"
|
16
|
+
|
17
|
+
spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
|
18
|
+
spec.bindir = "exe"
|
19
|
+
spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
|
20
|
+
spec.require_paths = ["lib"]
|
21
|
+
spec.test_files = spec.files.grep(%r{^(test|s|features)/})
|
22
|
+
|
23
|
+
spec.add_dependency 'capybara', '~> 2.10'
|
24
|
+
spec.add_dependency 'poltergeist', '~> 1.11'
|
25
|
+
spec.add_dependency 'phantomjs', '~> 2.1'
|
26
|
+
spec.add_dependency 'atomic', '~> 1.1'
|
27
|
+
spec.add_dependency 'minitest', '~> 5.8'
|
28
|
+
|
29
|
+
spec.add_development_dependency "bundler", "~> 1.10"
|
30
|
+
spec.add_development_dependency "rake", "~> 10.0"
|
31
|
+
end
|
@@ -0,0 +1,9 @@
|
|
1
|
+
require 'intransient_capybara/version'
|
2
|
+
require 'intransient_capybara/patches'
|
3
|
+
require 'intransient_capybara/minitest_retry'
|
4
|
+
require 'intransient_capybara/intransient_capybara_helper'
|
5
|
+
|
6
|
+
module IntransientCapybara
|
7
|
+
autoload :IntransientCapybaraTest, 'intransient_capybara/intransient_capybara_test'
|
8
|
+
autoload :RackRequestBlocker, 'intransient_capybara/rack_request_blocker'
|
9
|
+
end
|
@@ -0,0 +1,105 @@
|
|
1
|
+
module IntransientCapybaraHelper
|
2
|
+
|
3
|
+
def wait_for_response!(max_ajax_seconds = 3)
|
4
|
+
wait_for_page_load!
|
5
|
+
wait_for_ajax!(max_ajax_seconds)
|
6
|
+
end
|
7
|
+
|
8
|
+
def wait_for_ajax!(max_seconds = 3)
|
9
|
+
active_ajax_count = nil
|
10
|
+
|
11
|
+
begin
|
12
|
+
wait_for('ajax requests', max_wait_time: max_seconds, polling_interval: 0.1) do
|
13
|
+
page.document.synchronize do
|
14
|
+
(page.evaluate_script 'typeof($) != "undefined"') && (active_ajax_count = page.evaluate_script("$.active")).zero?
|
15
|
+
Rails.logger.info "#{active_ajax_count} outstanding XHR(s)"
|
16
|
+
end
|
17
|
+
end
|
18
|
+
rescue StandardError
|
19
|
+
assert(false, "waited #{max_seconds} seconds for ajax complete but #{active_ajax_count} ajax calls still active")
|
20
|
+
end
|
21
|
+
|
22
|
+
true
|
23
|
+
end
|
24
|
+
|
25
|
+
def wait_for_page_load!
|
26
|
+
page.document.synchronize do
|
27
|
+
current_path
|
28
|
+
true
|
29
|
+
end
|
30
|
+
end
|
31
|
+
|
32
|
+
def teardown_wait_for_requests_complete!
|
33
|
+
stop_client
|
34
|
+
block_rack_requests!
|
35
|
+
|
36
|
+
wait_for('pending AJAX requests complete') do
|
37
|
+
if ENV.fetch('TRACE_TEST_FRAMEWORK', false) == 'true'
|
38
|
+
puts 'I am waiting for rack requests to clear out...'
|
39
|
+
end
|
40
|
+
RackRequestBlocker.num_active_requests == 0
|
41
|
+
end
|
42
|
+
end
|
43
|
+
|
44
|
+
def block_rack_requests!
|
45
|
+
if ENV.fetch('TRACE_TEST_FRAMEWORK', false) == 'true'
|
46
|
+
puts 'I am turning off rack requests'
|
47
|
+
end
|
48
|
+
|
49
|
+
RackRequestBlocker.block_requests!
|
50
|
+
end
|
51
|
+
|
52
|
+
def allow_rack_requests!
|
53
|
+
if ENV.fetch('TRACE_TEST_FRAMEWORK', false) == 'true'
|
54
|
+
puts 'I am turning on rack requests'
|
55
|
+
end
|
56
|
+
|
57
|
+
RackRequestBlocker.allow_requests!
|
58
|
+
end
|
59
|
+
|
60
|
+
def wait_for(condition_name, max_wait_time: 30, polling_interval: 0.5)
|
61
|
+
wait_until = Time.now + max_wait_time.seconds
|
62
|
+
while true
|
63
|
+
return if yield
|
64
|
+
if Time.now > wait_until
|
65
|
+
raise "Condition not met: #{condition_name}"
|
66
|
+
else
|
67
|
+
sleep(polling_interval)
|
68
|
+
end
|
69
|
+
end
|
70
|
+
end
|
71
|
+
|
72
|
+
def stop_client
|
73
|
+
page.execute_script %Q{
|
74
|
+
window.location = "about:blank";
|
75
|
+
}
|
76
|
+
end
|
77
|
+
|
78
|
+
def report_traffic
|
79
|
+
if ENV.fetch('DEBUG_TEST_TRAFFIC', false) == 'true'
|
80
|
+
puts "Downloaded #{page.driver.network_traffic.map(&:response_parts).flatten.map(&:body_size).compact.sum / 1.megabyte} megabytes"
|
81
|
+
puts "Processed #{page.driver.network_traffic.size} network requests"
|
82
|
+
|
83
|
+
grouped_urls = page.driver.network_traffic.map(&:url).group_by{|url| /\Ahttps?:\/\/(?:.*\.)?(?:localhost|127\.0\.0\.1)/.match(url).present?}
|
84
|
+
internal_urls = grouped_urls[true]
|
85
|
+
external_urls = grouped_urls[false]
|
86
|
+
|
87
|
+
if internal_urls.present?
|
88
|
+
puts "Local URLs queried: #{internal_urls}"
|
89
|
+
end
|
90
|
+
|
91
|
+
if external_urls.present?
|
92
|
+
puts "External URLs queried: #{external_urls}"
|
93
|
+
|
94
|
+
if ENV.fetch('DEBUG_TEST_TRAFFIC_RAISE_EXTERNAL', false) == 'true'
|
95
|
+
raise "Queried external URLs! This will be slow! #{external_urls}"
|
96
|
+
end
|
97
|
+
end
|
98
|
+
end
|
99
|
+
end
|
100
|
+
|
101
|
+
def resize_window_by(size)
|
102
|
+
page.driver.resize size[0], size[1]
|
103
|
+
end
|
104
|
+
|
105
|
+
end
|
@@ -0,0 +1,121 @@
|
|
1
|
+
require "capybara/rails"
|
2
|
+
require "capybara/poltergeist"
|
3
|
+
require 'atomic'
|
4
|
+
|
5
|
+
class IntransientCapybaraTest < ActionDispatch::IntegrationTest
|
6
|
+
extend Minitest::OptionalRetry
|
7
|
+
|
8
|
+
include IntransientCapybaraHelper
|
9
|
+
include Capybara::DSL
|
10
|
+
|
11
|
+
cattr_accessor :blacklisted_urls
|
12
|
+
cattr_writer :default_window_size
|
13
|
+
cattr_writer :cache_warmup_path
|
14
|
+
|
15
|
+
@@warm_asset_cache = Atomic.new(false)
|
16
|
+
@@warming_asset_cache = Atomic.new(false)
|
17
|
+
|
18
|
+
Capybara.default_max_wait_time = 10
|
19
|
+
Capybara.current_driver = :poltergeist
|
20
|
+
Capybara.javascript_driver = :poltergeist
|
21
|
+
|
22
|
+
def setup
|
23
|
+
super
|
24
|
+
|
25
|
+
@setup_called = true
|
26
|
+
|
27
|
+
if ENV.fetch('TRACE_TEST_FRAMEWORK', false) == 'true'
|
28
|
+
puts 'I am in capybara setup method'
|
29
|
+
end
|
30
|
+
|
31
|
+
page.driver.browser.url_blacklist = self.class.blacklisted_urls
|
32
|
+
|
33
|
+
resize_window_by default_window_size
|
34
|
+
|
35
|
+
warm_asset_cache
|
36
|
+
|
37
|
+
allow_rack_requests!
|
38
|
+
end
|
39
|
+
|
40
|
+
def teardown
|
41
|
+
if ENV.fetch('TRACE_TEST_FRAMEWORK', false) == 'true'
|
42
|
+
puts 'I am in capybara teardown method'
|
43
|
+
end
|
44
|
+
|
45
|
+
@teardown_called = true
|
46
|
+
|
47
|
+
# Wait on outstanding requests (but tests should not do stuff at the end, why bother clicking stuff you don't assert on?)
|
48
|
+
# If you don't do this, the next time will fail because the server is busy
|
49
|
+
wait_for_response!
|
50
|
+
teardown_wait_for_requests_complete!
|
51
|
+
|
52
|
+
report_traffic
|
53
|
+
page.driver.clear_network_traffic
|
54
|
+
|
55
|
+
page.driver.clear_cookies
|
56
|
+
Capybara.reset_sessions!
|
57
|
+
|
58
|
+
super
|
59
|
+
end
|
60
|
+
|
61
|
+
protected
|
62
|
+
|
63
|
+
def after_teardown
|
64
|
+
super
|
65
|
+
|
66
|
+
unless @setup_called
|
67
|
+
raise 'Setup was not called in the parent! You MUST call super in your overrides!'
|
68
|
+
end
|
69
|
+
|
70
|
+
unless @teardown_called
|
71
|
+
raise 'Teardown was not called in the parent! You MUST call super in your overrides!'
|
72
|
+
end
|
73
|
+
end
|
74
|
+
|
75
|
+
def default_window_size
|
76
|
+
@@default_window_size || [1024, 768]
|
77
|
+
end
|
78
|
+
|
79
|
+
def cache_warmup_path
|
80
|
+
@@cache_warmup_path || '/'
|
81
|
+
end
|
82
|
+
|
83
|
+
def warm_asset_cache
|
84
|
+
return if warm_asset_cache?
|
85
|
+
|
86
|
+
if warming_asset_cache?
|
87
|
+
wait_for('assets to be warmed up', max_wait_time: 60, polling_interval: 3) do
|
88
|
+
warm_asset_cache?
|
89
|
+
end
|
90
|
+
end
|
91
|
+
|
92
|
+
@@warming_asset_cache.value = true
|
93
|
+
puts 'WARMING UP ASSET CACHE...'
|
94
|
+
|
95
|
+
begin
|
96
|
+
allow_rack_requests!
|
97
|
+
visit cache_warmup_path
|
98
|
+
sleep 15
|
99
|
+
wait_for_response!
|
100
|
+
teardown_wait_for_requests_complete!
|
101
|
+
|
102
|
+
@@warm_asset_cache.value = true
|
103
|
+
rescue StandardError => e
|
104
|
+
puts "Could not warm asset cache - #{e.class.to_s} - #{e.message}"
|
105
|
+
return
|
106
|
+
ensure
|
107
|
+
@@warming_asset_cache.value = false
|
108
|
+
end
|
109
|
+
|
110
|
+
puts 'ASSETS HOT N READY!'
|
111
|
+
end
|
112
|
+
|
113
|
+
def warming_asset_cache?
|
114
|
+
@@warming_asset_cache.value
|
115
|
+
end
|
116
|
+
|
117
|
+
def warm_asset_cache?
|
118
|
+
ENV.fetch('TEST_SKIP_WARMING_ASSET_CACHE', false) == 'true' || @@warm_asset_cache.value
|
119
|
+
end
|
120
|
+
|
121
|
+
end
|
@@ -0,0 +1,54 @@
|
|
1
|
+
# Adapted from https://github.com/appfolio/minitest-optional_retry/blob/master/lib/minitest/optional_retry.rb
|
2
|
+
require 'minitest/autorun'
|
3
|
+
require 'atomic'
|
4
|
+
|
5
|
+
module Minitest
|
6
|
+
module OptionalRetry
|
7
|
+
|
8
|
+
GLOBAL_RETRY_LIMIT = 25
|
9
|
+
@@global_retry_count = Atomic.new(0)
|
10
|
+
|
11
|
+
def run_one_method(klass, method_name, reporter)
|
12
|
+
first_result = nil
|
13
|
+
report_result = nil
|
14
|
+
|
15
|
+
retries = [ENV.fetch('MINITEST_RETRY_COUNT', 3).to_i, 1].max
|
16
|
+
|
17
|
+
retries.times do
|
18
|
+
result = Minitest.run_one_method(klass, method_name)
|
19
|
+
|
20
|
+
first_result ||= result
|
21
|
+
|
22
|
+
if result.passed?
|
23
|
+
unless ENV.fetch('TRANSIENT_TESTS_REPORT_FAILURE', false) == 'true'
|
24
|
+
report_result = result
|
25
|
+
end
|
26
|
+
break
|
27
|
+
else
|
28
|
+
if @@global_retry_count.value >= GLOBAL_RETRY_LIMIT
|
29
|
+
puts 'We hit the global retry limit, you probably have a more global issue going on. We will not try to rerun transients anymore.'
|
30
|
+
break
|
31
|
+
end
|
32
|
+
|
33
|
+
if result.failure.exception.present? && result.failure.exception.class.to_s == 'Capybara::Poltergeist::DeadClient'
|
34
|
+
puts "PhantomJS died!!!! - #{klass.to_s}##{method_name}"
|
35
|
+
next
|
36
|
+
end
|
37
|
+
|
38
|
+
puts "Test failed!!!! - #{klass.to_s}##{method_name}"
|
39
|
+
|
40
|
+
@@global_retry_count.update { |v| v + 1 }
|
41
|
+
end
|
42
|
+
|
43
|
+
end
|
44
|
+
|
45
|
+
report_result ||= first_result
|
46
|
+
|
47
|
+
if !first_result.passed? && report_result.passed?
|
48
|
+
puts "#{klass.to_s}##{method_name} IS TRANSIENT!!!!! IT FAILED THE FIRST TRY THEN PASSED IN RETRIES"
|
49
|
+
end
|
50
|
+
|
51
|
+
reporter.record(report_result)
|
52
|
+
end
|
53
|
+
end
|
54
|
+
end
|
@@ -0,0 +1,61 @@
|
|
1
|
+
module Capybara
|
2
|
+
module Node
|
3
|
+
class Base
|
4
|
+
def synchronize(seconds=Capybara.default_max_wait_time, options = {})
|
5
|
+
start_time = Capybara::Helpers.monotonic_time
|
6
|
+
|
7
|
+
begin
|
8
|
+
if session.synchronized
|
9
|
+
yield
|
10
|
+
else
|
11
|
+
session.synchronized = true
|
12
|
+
begin
|
13
|
+
yield
|
14
|
+
rescue => e
|
15
|
+
session.raise_server_error!
|
16
|
+
raise e unless driver.wait?
|
17
|
+
raise e unless catch_error?(e, options[:errors])
|
18
|
+
if (Capybara::Helpers.monotonic_time - start_time) >= seconds
|
19
|
+
|
20
|
+
### This is a patched part
|
21
|
+
warn "Capybara's timeout limit reached - if your tests are green, something is wrong"
|
22
|
+
###
|
23
|
+
|
24
|
+
raise e
|
25
|
+
end
|
26
|
+
sleep(0.05)
|
27
|
+
raise Capybara::FrozenInTime, "time appears to be frozen, Capybara does not work with libraries which freeze time, consider using time travelling instead" if Capybara::Helpers.monotonic_time == start_time
|
28
|
+
reload if Capybara.automatic_reload
|
29
|
+
retry
|
30
|
+
ensure
|
31
|
+
session.synchronized = false
|
32
|
+
end
|
33
|
+
end
|
34
|
+
rescue Capybara::ElementNotFound => sad_day
|
35
|
+
### This is a patched part
|
36
|
+
|
37
|
+
puts 'We failed to find an element :('
|
38
|
+
puts
|
39
|
+
puts 'We are on path: ' + session.current_path
|
40
|
+
puts
|
41
|
+
puts 'This is the page HTML: ' + session.body
|
42
|
+
puts
|
43
|
+
raise sad_day
|
44
|
+
|
45
|
+
###
|
46
|
+
end
|
47
|
+
|
48
|
+
end
|
49
|
+
end
|
50
|
+
end
|
51
|
+
|
52
|
+
class Session
|
53
|
+
def raise_server_error!
|
54
|
+
if Capybara.raise_server_errors and @server and @server.error
|
55
|
+
raise @server.error
|
56
|
+
end
|
57
|
+
ensure
|
58
|
+
@server.reset_error! if @server
|
59
|
+
end
|
60
|
+
end
|
61
|
+
end
|
@@ -0,0 +1,57 @@
|
|
1
|
+
require 'atomic'
|
2
|
+
|
3
|
+
# Rack middleware that keeps track of the number of active requests and can block new requests.
|
4
|
+
class RackRequestBlocker
|
5
|
+
|
6
|
+
@@num_active_requests = Atomic.new(0)
|
7
|
+
@@block_requests = Atomic.new(false)
|
8
|
+
|
9
|
+
# Returns the number of requests the server is currently processing.
|
10
|
+
def self.num_active_requests
|
11
|
+
@@num_active_requests.value
|
12
|
+
end
|
13
|
+
|
14
|
+
# Prevents the server from accepting new requests. Any new requests will return an HTTP
|
15
|
+
# 503 status.
|
16
|
+
def self.block_requests!
|
17
|
+
@@block_requests.value = true
|
18
|
+
end
|
19
|
+
|
20
|
+
# Allows the server to accept requests again.
|
21
|
+
def self.allow_requests!
|
22
|
+
@@block_requests.value = false
|
23
|
+
end
|
24
|
+
|
25
|
+
def initialize(app)
|
26
|
+
@app = app
|
27
|
+
end
|
28
|
+
|
29
|
+
def call(env)
|
30
|
+
increment_active_requests
|
31
|
+
if block_requests?
|
32
|
+
block_request(env)
|
33
|
+
else
|
34
|
+
@app.call(env)
|
35
|
+
end
|
36
|
+
ensure
|
37
|
+
decrement_active_requests
|
38
|
+
end
|
39
|
+
|
40
|
+
private
|
41
|
+
|
42
|
+
def block_requests?
|
43
|
+
@@block_requests.value
|
44
|
+
end
|
45
|
+
|
46
|
+
def block_request(env)
|
47
|
+
[503, {}, []]
|
48
|
+
end
|
49
|
+
|
50
|
+
def increment_active_requests
|
51
|
+
@@num_active_requests.update { |v| v + 1 }
|
52
|
+
end
|
53
|
+
|
54
|
+
def decrement_active_requests
|
55
|
+
@@num_active_requests.update { |v| v - 1 }
|
56
|
+
end
|
57
|
+
end
|
metadata
ADDED
@@ -0,0 +1,159 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: intransient_capybara
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 1.0.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Seth Ringling
|
8
|
+
autorequire:
|
9
|
+
bindir: exe
|
10
|
+
cert_chain: []
|
11
|
+
date: 2016-11-29 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: capybara
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - "~>"
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: '2.10'
|
20
|
+
type: :runtime
|
21
|
+
prerelease: false
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - "~>"
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: '2.10'
|
27
|
+
- !ruby/object:Gem::Dependency
|
28
|
+
name: poltergeist
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - "~>"
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: '1.11'
|
34
|
+
type: :runtime
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - "~>"
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: '1.11'
|
41
|
+
- !ruby/object:Gem::Dependency
|
42
|
+
name: phantomjs
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - "~>"
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '2.1'
|
48
|
+
type: :runtime
|
49
|
+
prerelease: false
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - "~>"
|
53
|
+
- !ruby/object:Gem::Version
|
54
|
+
version: '2.1'
|
55
|
+
- !ruby/object:Gem::Dependency
|
56
|
+
name: atomic
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - "~>"
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '1.1'
|
62
|
+
type: :runtime
|
63
|
+
prerelease: false
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - "~>"
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: '1.1'
|
69
|
+
- !ruby/object:Gem::Dependency
|
70
|
+
name: minitest
|
71
|
+
requirement: !ruby/object:Gem::Requirement
|
72
|
+
requirements:
|
73
|
+
- - "~>"
|
74
|
+
- !ruby/object:Gem::Version
|
75
|
+
version: '5.8'
|
76
|
+
type: :runtime
|
77
|
+
prerelease: false
|
78
|
+
version_requirements: !ruby/object:Gem::Requirement
|
79
|
+
requirements:
|
80
|
+
- - "~>"
|
81
|
+
- !ruby/object:Gem::Version
|
82
|
+
version: '5.8'
|
83
|
+
- !ruby/object:Gem::Dependency
|
84
|
+
name: bundler
|
85
|
+
requirement: !ruby/object:Gem::Requirement
|
86
|
+
requirements:
|
87
|
+
- - "~>"
|
88
|
+
- !ruby/object:Gem::Version
|
89
|
+
version: '1.10'
|
90
|
+
type: :development
|
91
|
+
prerelease: false
|
92
|
+
version_requirements: !ruby/object:Gem::Requirement
|
93
|
+
requirements:
|
94
|
+
- - "~>"
|
95
|
+
- !ruby/object:Gem::Version
|
96
|
+
version: '1.10'
|
97
|
+
- !ruby/object:Gem::Dependency
|
98
|
+
name: rake
|
99
|
+
requirement: !ruby/object:Gem::Requirement
|
100
|
+
requirements:
|
101
|
+
- - "~>"
|
102
|
+
- !ruby/object:Gem::Version
|
103
|
+
version: '10.0'
|
104
|
+
type: :development
|
105
|
+
prerelease: false
|
106
|
+
version_requirements: !ruby/object:Gem::Requirement
|
107
|
+
requirements:
|
108
|
+
- - "~>"
|
109
|
+
- !ruby/object:Gem::Version
|
110
|
+
version: '10.0'
|
111
|
+
description: With improved debuggability, with proper usage and configuration of Capybara/Poltergeist/PhantomJS,
|
112
|
+
and with some improvements on top of it we can greatly reduce the occurrence of
|
113
|
+
transient integration/UI test failures.
|
114
|
+
email:
|
115
|
+
- sringling@avvo.com
|
116
|
+
executables: []
|
117
|
+
extensions: []
|
118
|
+
extra_rdoc_files: []
|
119
|
+
files:
|
120
|
+
- ".gitignore"
|
121
|
+
- Gemfile
|
122
|
+
- README.md
|
123
|
+
- Rakefile
|
124
|
+
- bin/console
|
125
|
+
- bin/setup
|
126
|
+
- intransient_capybara.gemspec
|
127
|
+
- lib/intransient_capybara.rb
|
128
|
+
- lib/intransient_capybara/intransient_capybara_helper.rb
|
129
|
+
- lib/intransient_capybara/intransient_capybara_test.rb
|
130
|
+
- lib/intransient_capybara/minitest_retry.rb
|
131
|
+
- lib/intransient_capybara/patches.rb
|
132
|
+
- lib/intransient_capybara/rack_request_blocker.rb
|
133
|
+
- lib/intransient_capybara/version.rb
|
134
|
+
homepage: https://github.com/avvo/intransient_capybara
|
135
|
+
licenses: []
|
136
|
+
metadata: {}
|
137
|
+
post_install_message:
|
138
|
+
rdoc_options: []
|
139
|
+
require_paths:
|
140
|
+
- lib
|
141
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
142
|
+
requirements:
|
143
|
+
- - ">="
|
144
|
+
- !ruby/object:Gem::Version
|
145
|
+
version: '0'
|
146
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
147
|
+
requirements:
|
148
|
+
- - ">="
|
149
|
+
- !ruby/object:Gem::Version
|
150
|
+
version: '0'
|
151
|
+
requirements: []
|
152
|
+
rubyforge_project:
|
153
|
+
rubygems_version: 2.4.8
|
154
|
+
signing_key:
|
155
|
+
specification_version: 4
|
156
|
+
summary: A set of improvements to Capybara/Poltergeist/PhantomJS test stack that reduces
|
157
|
+
the occurrence transient failures.
|
158
|
+
test_files: []
|
159
|
+
has_rdoc:
|