paraspec 0.0.2 → 0.0.3

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 76dfd75abb077976992e37c9934206e610f652f8f7d88564bcfdd9356cf473e1
4
- data.tar.gz: 5d08c00a5d951348ba228bb94509158ddae389b21e0e500b011cbdb6aa379a52
3
+ metadata.gz: d6a3db1f3be0ea3c3b137d2d22ed55d29e402a94f72123c2c5d04eb5c3560569
4
+ data.tar.gz: 25ea9c3f19e0802ca3db168c0090d1283f5db337477fd42dbcb993f9ccbf2c0a
5
5
  SHA512:
6
- metadata.gz: e04a18244f08e43e84a9a610405fa6972820f658b84ba67e81a954fd5635873aba6e53ddef8277985d124dbcb6ec6055e5dfbe7f51930c92e314c44078869aeb
7
- data.tar.gz: 8b7ef16dee8bb3506442cb8a7b761dbb91929881acd74c4369203f390381de0a3902d59341ce58eb23fa41684754bbd911bb8f67b3ba8a80eddd3fa379689aa4
6
+ metadata.gz: 6f71a077b52ac683f766e15104204fb27e25c276d08d203f27d3eae7171f5d52ddf4003951f1a8185042d9c382065c2e5d3824719462efb8eb7897063cb978db
7
+ data.tar.gz: 8cf4dece33abe87139dc218ad1848f30d4ed02b2e60df98005aa7f6c2253b96f42c9eb1b16cfb782ca903328f275c6823a83c737277df1c946956c6055928bdb
data/README.md CHANGED
@@ -18,7 +18,8 @@ test by test basis, avoiding interleaving output of different tests together.
18
18
  This output capture can be performed for output generated by C extensions
19
19
  as well as plain Ruby code.
20
20
  4. Test results are seamlessly integrated by the master, such that
21
- a parallel run produces a single progress bar with Fuubar across all workers.
21
+ a parallel run produces a single progress bar with
22
+ [Fuubar](https://github.com/thekompanee/fuubar) across all workers.
22
23
  5. Test grouping: paraspec can[**] group the tests by file, top level example
23
24
  group, bottom level example group and individually by examples.
24
25
  Larger groupings reduce overhead but limit concurrency.
@@ -33,6 +34,27 @@ is not currently on the roadmap.
33
34
 
34
35
  [**] Currently only grouping by bottom level example group is implemented.
35
36
 
37
+ ## Performance
38
+
39
+ How much of a difference does paraspec make? The answer, as one might
40
+ expect, varies greatly with the test suite being run as well as available
41
+ hardware. Here are some examples:
42
+
43
+ | Example | Hardware | Sequential | Paraspec (c=2) | Paraspec (c=4) |
44
+ |---------|------------|----------------|----------------|----------|
45
+ | [MongoDB Ruby Driver](https://docs.mongodb.com/ruby-driver/current/) test suite | Travis CI | 16 minutes | 13 minutes | 10-11 minutes |
46
+ | [MongoDB Ruby Driver](https://docs.mongodb.com/ruby-driver/current/) test suite | 14-core workstation | 15 minutes | | 4 minutes |
47
+
48
+ [Exampe Travis build](https://travis-ci.org/p-mongo/mongo-ruby-driver-paraspec/builds/411986888)
49
+
50
+ Even on Travis, which is likely limited to a single core, using 4x concurrency
51
+ reduces the runtime by 5 minutes. On a developer workstation which doesn't
52
+ download binaries on every test run the speedup is closer to linear.
53
+ Waiting 4 minutes instead of 15 for a complete test suite means the engineers
54
+ can actually run the complete test suite as part of their normal workflow,
55
+ instead of sending the code to a CI platform and context switching to
56
+ a different project.
57
+
36
58
  ## Usage
37
59
 
38
60
  Add paraspec to your Gemfile:
@@ -88,7 +110,8 @@ failing tests.
88
110
 
89
111
  ### Debugging
90
112
 
91
- Paraspec offers several debugging aids. The first one is the terminal option:
113
+ Paraspec offers several debugging aids. For interactive debugging use the
114
+ terminal option:
92
115
 
93
116
  paraspec -T
94
117
 
@@ -108,6 +131,34 @@ The debugging output is turned on with `-d`/`--debug` option:
108
131
  paraspec -d ipc # IPC requests and responses
109
132
  paraspec -d perf # timing & performance information
110
133
 
134
+ ### Executing Tests Together
135
+
136
+ It is possible to specify that a group of tests should be executed in the
137
+ same worker rather than distributed. This is useful when the setup for
138
+ the tests is expensive and therefore is done once with multiple tests
139
+ defined on the result.
140
+
141
+ The grouping is defined by specifying `{group: true}` paraspec option on
142
+ a `describe` or a `context` block as follows:
143
+
144
+ describe 'Run these together', paraspec: {group: true} do
145
+ before(:all) do
146
+ # expensive setup
147
+ end
148
+
149
+ it 'does something' do
150
+ # ...
151
+ end
152
+
153
+ it 'does something else' do
154
+ # ...
155
+ end
156
+
157
+ after(:all) do
158
+ # teardown
159
+ end
160
+ end
161
+
111
162
  ## Caveats
112
163
 
113
164
  The master and workers need to all define the same example groups and
@@ -16,7 +16,7 @@ OptionParser.new do |opts|
16
16
  options[:concurrency] = v.to_i
17
17
  end
18
18
 
19
- opts.on('-d', '--debug=SUBSYSTEM', 'Output debugging information for SUBSYSTEM') do |v|
19
+ opts.on('-d', '--debug=SUBSYSTEM', 'Output debugging information for SUBSYSTEM (state/ipc/perf)') do |v|
20
20
  options[:"debug_#{v}"] = true
21
21
  options[:debug] = true
22
22
  end
@@ -44,7 +44,7 @@ end
44
44
  RSpec.configuration.files_or_directories_to_run = files
45
45
  =end
46
46
 
47
- %w(ipc state perf).each do |subsystem|
47
+ %w(state ipc perf).each do |subsystem|
48
48
  if options.delete(:"debug_#{subsystem}")
49
49
  Paraspec.logger.send("log_#{subsystem}=", true)
50
50
  end
@@ -1,5 +1,6 @@
1
1
  require 'paraspec/rspec_patches'
2
2
  require 'paraspec/version'
3
+ require 'paraspec/errors'
3
4
  require 'paraspec/logger'
4
5
  require 'paraspec/ipc'
5
6
  require 'paraspec/drb_helpers'
@@ -0,0 +1,13 @@
1
+ module Paraspec
2
+ # Base class for all Paraspec errors
3
+ class Error < StandardError; end
4
+
5
+ # Different set of examples between master and a worker
6
+ class InconsistentTestSuite < Error; end
7
+
8
+ # A worker process exited with non-zero status
9
+ class WorkerFailed < StandardError; end
10
+
11
+ # Master process exited with non-zero status
12
+ class MasterFailed < StandardError; end
13
+ end
@@ -6,7 +6,7 @@ module Paraspec
6
6
  def initialize(options={})
7
7
  @terminal = options[:terminal]
8
8
 
9
- @client = Faraday.new(url: "http://localhost:#{Paraspec::MASTER_APP_PORT}") do |client|
9
+ @client = Faraday.new(url: "http://localhost:#{Paraspec::Ipc.master_app_port}") do |client|
10
10
  client.adapter :net_http
11
11
  if @terminal
12
12
  client.options.timeout = 100000
@@ -1,11 +1,26 @@
1
1
  require 'hashie'
2
+ require 'socket'
2
3
 
3
4
  module Paraspec
4
- #SUPERVISOR_DRB_URI = "druby://localhost:6030"
5
- MASTER_DRB_URI = "druby://localhost:6031"
6
- MASTER_APP_PORT = 6031
7
-
8
5
  class IpcHash < Hash
9
6
  include Hashie::Extensions::IndifferentAccess
10
7
  end
8
+
9
+ module Ipc
10
+ class << self
11
+ def pick_master_app_port
12
+ port = 10000 + rand(40000)
13
+ begin
14
+ server = TCPServer.new('127.0.0.1', port)
15
+ server.close
16
+ return @master_app_port = port
17
+ rescue Errno::EADDRINUSE
18
+ port = 10000 + rand(40000)
19
+ retry
20
+ end
21
+ end
22
+
23
+ attr_reader :master_app_port
24
+ end
25
+ end
11
26
  end
@@ -58,7 +58,7 @@ module Paraspec
58
58
  end
59
59
  @queue = []
60
60
  if @non_example_exception_count == 0
61
- @queue += RSpecFacade.all_example_groups
61
+ @queue += RSpecFacade.queueable_example_groups
62
62
  puts "#{@queue.length} example groups queued"
63
63
  else
64
64
  puts "#{@non_example_exception_count} errors outside of examples, aborting"
@@ -104,13 +104,20 @@ module Paraspec
104
104
  example_group = @queue.shift
105
105
  return nil if example_group.nil?
106
106
 
107
- # TODO I am still not 100% on what should be filtered and pruned where,
108
- # but we shouldn't be returning a specification here unless
109
- # there are tests in it that a worker will run
110
- pruned_examples = RSpec.configuration.filter_manager.prune(example_group.examples)
111
- next if pruned_examples.empty?
107
+ if example_group.metadata[:paraspec] && example_group.metadata[:paraspec][:group]
108
+ # unsplittable example group
109
+ # pass to worker as is and have the worker prune examples
110
+ m = example_group.metadata
111
+ else
112
+ # TODO I am still not 100% on what should be filtered and pruned where,
113
+ # but we shouldn't be returning a specification here unless
114
+ # there are tests in it that a worker will run
115
+ pruned_examples = RSpec.configuration.filter_manager.prune(example_group.examples)
116
+ next if pruned_examples.empty?
117
+
118
+ m = example_group.metadata
119
+ end
112
120
 
113
- m = example_group.metadata
114
121
  return {
115
122
  file_path: m[:file_path],
116
123
  scoped_id: m[:scoped_id],
@@ -39,7 +39,7 @@ module Paraspec
39
39
  private def connect
40
40
  start_time = Time.now
41
41
  begin
42
- @socket = TCPSocket.new('127.0.0.1', MASTER_APP_PORT)
42
+ @socket = TCPSocket.new('127.0.0.1', Paraspec::Ipc.master_app_port)
43
43
  rescue Errno::ECONNREFUSED
44
44
  if !@terminal && Time.now - start_time > DrbHelpers::WAIT_TIME
45
45
  raise
@@ -8,10 +8,11 @@ module Paraspec
8
8
 
9
9
  def initialize(master)
10
10
  @master = master
11
+ @lock = Mutex.new
11
12
  end
12
13
 
13
14
  def run
14
- @socket = ::TCPServer.new('127.0.0.1', MASTER_APP_PORT)
15
+ @socket = ::TCPServer.new('127.0.0.1', Paraspec::Ipc.master_app_port)
15
16
  begin
16
17
  while true
17
18
  s = @socket.accept_nonblock
@@ -29,30 +30,46 @@ module Paraspec
29
30
  Thread.new do
30
31
  u = unpacker(s)
31
32
  u.each do |obj|
32
- result = nil
33
- time = Benchmark.realtime do
34
- action = obj['action'].gsub('-', '_')
35
- payload = obj['payload']
36
- if payload
37
- payload = IpcHash.new.merge(payload)
38
- args = [payload]
39
- else
40
- args = []
41
- end
33
+ # Apparently this block may be run concurrently, which we are not
34
+ # equipped for.
35
+ @lock.synchronize do
36
+ process_request(s, obj)
37
+ end
38
+ end
39
+ end
40
+ end
42
41
 
43
- Paraspec.logger.debug_ipc("SrvReq:#{obj['id']} #{obj}")
44
- result = @master.send(action, *args)
42
+ def process_request(s, obj)
43
+ result = nil
44
+ time = Benchmark.realtime do
45
+ action = obj['action'].gsub('-', '_')
46
+ payload = obj['payload']
47
+ if payload
48
+ payload = IpcHash.new.merge(payload)
49
+ args = [payload]
50
+ else
51
+ args = []
52
+ end
45
53
 
46
- pk = packer(s)
47
- resp = {result: result}
48
- Paraspec.logger.debug_ipc("SrvRes:#{obj['id']} #{resp}")
49
- pk.write(resp)
50
- pk.flush
51
- s.flush
52
- end
53
- Paraspec.logger.debug_perf("SrvReq:#{obj['id']} #{obj['action']}: #{result} #{'%.3f msec' % (time*1000)}")
54
+ Paraspec.logger.debug_ipc("SrvReq:#{obj['id']} #{obj}")
55
+ result = @master.send(action, *args)
56
+
57
+ pk = packer(s)
58
+ resp = {result: result}
59
+ Paraspec.logger.debug_ipc("SrvRes:#{obj['id']} #{resp}")
60
+ if payload && payload['_noret']
61
+ # Requester does not want a response.
62
+ # We return a hash here because deserializer expects a hash.
63
+ # Always returning `resp` is problematic because it can
64
+ # be non-serializable.
65
+ pk.write({})
66
+ else
67
+ pk.write(resp)
54
68
  end
69
+ pk.flush
70
+ s.flush
55
71
  end
72
+ Paraspec.logger.debug_perf("SrvReq:#{obj['id']} #{obj['action']}: #{result} #{'%.3f msec' % (time*1000)}")
56
73
  end
57
74
  end
58
75
  end
@@ -7,7 +7,27 @@ module Paraspec
7
7
 
8
8
  class << self
9
9
  extend Forwardable
10
- def_delegators :instance, :all_example_groups, :all_examples
10
+ def_delegators :instance, :queueable_example_groups, :all_example_groups, :all_examples
11
+ end
12
+
13
+ def queueable_example_groups
14
+ @queueable_example_groups ||= begin
15
+ groups = [] + RSpec.world.example_groups
16
+ all_groups = []
17
+ until groups.empty?
18
+ new_groups = []
19
+ groups.each do |group|
20
+ all_groups << group
21
+ if group.metadata[:paraspec] && group.metadata[:paraspec][:group]
22
+ # unsplittable group
23
+ else
24
+ new_groups += group.children
25
+ end
26
+ end
27
+ groups = new_groups
28
+ end
29
+ all_groups
30
+ end
11
31
  end
12
32
 
13
33
  def all_example_groups
@@ -21,6 +21,14 @@ module Paraspec
21
21
  Process.setpgrp
22
22
  end
23
23
 
24
+ at_exit do
25
+ # It seems that when the tests are run in Travis, killing
26
+ # paraspec vaporizes the standard output... flush the
27
+ # standard output streams here to work around.
28
+ STDERR.flush
29
+ STDOUT.flush
30
+ end
31
+
24
32
  supervisor_pid = $$
25
33
  at_exit do
26
34
  # We fork, therefore this handler will be run in master and
@@ -41,9 +49,11 @@ module Paraspec
41
49
  end
42
50
  end
43
51
 
52
+ Paraspec::Ipc.pick_master_app_port
53
+
44
54
  rd, wr = IO.pipe
45
55
  if @master_pid = fork
46
- # parent
56
+ # parent - supervisor
47
57
  wr.close
48
58
  @master_pipe = rd
49
59
  run_supervisor
@@ -93,7 +103,7 @@ module Paraspec
93
103
  Paraspec.logger.debug_state("Waiting for workers")
94
104
  @worker_pids.each_with_index do |pid, i|
95
105
  Paraspec.logger.debug_state("Waiting for worker #{i+1} at #{pid}")
96
- wait_for_process(pid)
106
+ wait_for_process(pid, "Worker #{i+1}", WorkerFailed)
97
107
  end
98
108
  status = 0
99
109
  else
@@ -108,15 +118,14 @@ module Paraspec
108
118
  end
109
119
  Paraspec.logger.debug_state("Asking master to stop")
110
120
  master_client.request('stop')
111
- wait_for_process(@master_pid)
121
+ wait_for_process(@master_pid, 'Master', MasterFailed)
112
122
  exit status
113
123
  end
114
124
 
115
- def wait_for_process(pid)
116
- begin
117
- Process.wait(pid)
118
- rescue Errno::ECHILD
119
- # already dead
125
+ def wait_for_process(pid, process_name, exception_class)
126
+ pid, status = Process.wait2(pid)
127
+ if status.exitstatus != 0
128
+ raise exception_class, "#{process_name} exited with status #{status.exitstatus}"
120
129
  end
121
130
  end
122
131
 
@@ -1,3 +1,3 @@
1
1
  module Paraspec
2
- VERSION = '0.0.2'
2
+ VERSION = '0.0.3'
3
3
  end
@@ -46,7 +46,7 @@ module Paraspec
46
46
  # being run.
47
47
  puts "Worker #{@number} has #{RSpecFacade.all_examples.count} examples, master has #{master_example_count}"
48
48
  #byebug
49
- raise "Worker #{@number} has #{RSpecFacade.all_examples.count} examples, master has #{master_example_count}"
49
+ raise InconsistentTestSuite, "Worker #{@number} has #{RSpecFacade.all_examples.count} examples, master has #{master_example_count}"
50
50
  end
51
51
 
52
52
  while true
@@ -54,7 +54,7 @@ module Paraspec
54
54
  file_path: notification.example.metadata[:file_path],
55
55
  scoped_id: notification.example.metadata[:scoped_id],
56
56
  }
57
- @master_client.request('notify-example-started', spec: spec)
57
+ @master_client.request('notify-example-started', spec: spec, _noret: true)
58
58
  end
59
59
 
60
60
  def example_passed(notification)
@@ -68,7 +68,7 @@ module Paraspec
68
68
  }
69
69
  execution_result = notification.example.execution_result
70
70
  @master_client.request('example-passed',
71
- spec: spec, result: execution_result)
71
+ spec: spec, result: execution_result, _noret: true)
72
72
  end
73
73
 
74
74
  def example_failed(notification)
@@ -29,14 +29,30 @@ module Paraspec
29
29
  #byebug
30
30
  raise "No example group for #{spec.inspect}"
31
31
  end
32
- examples = group.examples
33
- #Paraspec.logger.debug_state("Spec #{spec}: #{examples.length} examples")
34
- return if examples.empty?
35
- ids = examples.map { |e| e.metadata[:scoped_id] }
36
- RSpec.configuration.send(:instance_variable_set, '@filter_manager', RSpec::Core::FilterManager.new)
37
- RSpec.configuration.filter_manager.add_ids(spec[:file_path], ids)
38
- RSpec.world.filter_examples
39
- examples = RSpec.configuration.filter_manager.prune(examples)
32
+ if group.metadata[:paraspec] && group.metadata[:paraspec][:group]
33
+ # unsplittable group
34
+ # get all examples in child groups
35
+ examples = []
36
+ group_queue = [group]
37
+ until group_queue.empty?
38
+ next_group_queue = []
39
+ group_queue.each do |group|
40
+ next_group_queue += group.children
41
+ examples += group.examples
42
+ end
43
+ group_queue = next_group_queue
44
+ end
45
+ else
46
+ # leaf group
47
+ examples = group.examples
48
+ #Paraspec.logger.debug_state("Spec #{spec}: #{examples.length} examples")
49
+ return if examples.empty?
50
+ ids = examples.map { |e| e.metadata[:scoped_id] }
51
+ RSpec.configuration.send(:instance_variable_set, '@filter_manager', RSpec::Core::FilterManager.new)
52
+ RSpec.configuration.filter_manager.add_ids(spec[:file_path], ids)
53
+ RSpec.world.filter_examples
54
+ examples = RSpec.configuration.filter_manager.prune(examples)
55
+ end
40
56
  return if examples.empty?
41
57
  # It is important to run the entire world here because if
42
58
  # a particular example group is run, before/after :all hooks
metadata CHANGED
@@ -1,29 +1,35 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: paraspec
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.2
4
+ version: 0.0.3
5
5
  platform: ruby
6
6
  authors:
7
7
  - Oleg Pudeyev
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2018-08-04 00:00:00.000000000 Z
11
+ date: 2019-08-12 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: rspec-core
15
15
  requirement: !ruby/object:Gem::Requirement
16
16
  requirements:
17
- - - "~>"
17
+ - - ">="
18
18
  - !ruby/object:Gem::Version
19
19
  version: 3.7.1
20
+ - - "<"
21
+ - !ruby/object:Gem::Version
22
+ version: '4.0'
20
23
  type: :runtime
21
24
  prerelease: false
22
25
  version_requirements: !ruby/object:Gem::Requirement
23
26
  requirements:
24
- - - "~>"
27
+ - - ">="
25
28
  - !ruby/object:Gem::Version
26
29
  version: 3.7.1
30
+ - - "<"
31
+ - !ruby/object:Gem::Version
32
+ version: '4.0'
27
33
  - !ruby/object:Gem::Dependency
28
34
  name: childprocess
29
35
  requirement: !ruby/object:Gem::Requirement
@@ -78,6 +84,7 @@ files:
78
84
  - bin/paraspec
79
85
  - lib/paraspec.rb
80
86
  - lib/paraspec/drb_helpers.rb
87
+ - lib/paraspec/errors.rb
81
88
  - lib/paraspec/http_client.rb
82
89
  - lib/paraspec/http_server.rb
83
90
  - lib/paraspec/ipc.rb
@@ -115,9 +122,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
115
122
  - !ruby/object:Gem::Version
116
123
  version: '0'
117
124
  requirements: []
118
- rubyforge_project:
119
- rubygems_version: 2.7.3
125
+ rubygems_version: 3.0.3
120
126
  signing_key:
121
127
  specification_version: 4
122
- summary: paraspec-0.0.2
128
+ summary: paraspec-0.0.3
123
129
  test_files: []