in-parallel 0.1.9 → 0.1.10

Sign up to get free protection for your applications and to get access to all the features.
Files changed (7) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +41 -32
  3. data/in_parallel/version.rb +1 -1
  4. data/lib/in_parallel.rb +121 -81
  5. data/lib/parallel_enumerable.rb +9 -10
  6. data/lib/parallel_logger.rb +13 -0
  7. metadata +3 -2
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: a6277b0e376bda7d26d7f16760f16764569bdf01
4
- data.tar.gz: f0798ecc56cd7e02cd1784fbc28f4b19a3e76118
3
+ metadata.gz: 616a0cfcbc49b019d69c306e9e919a86efdd9fee
4
+ data.tar.gz: 4e7aba20c6ca158d124242c74ce44ca59fbf9179
5
5
  SHA512:
6
- metadata.gz: b3cb0370b0254ce1a801b32d7fcab2bb80acfeec6d0253a8752560c1da8ddf53de94e8a71d3e61369d204be44617451d26f909126ee484b61d0302c1b47d377d
7
- data.tar.gz: 2f769c82a0fb813eb3ebfbeb0f09a8fe1a00086ea1e0220bca5913c1c47d5a617d625070627d9e7a619fdb42aeb9a370c9a6aedb53b333111fee2d47840644cc
6
+ metadata.gz: d9e961955959052dd40268155c142deefa2950145519d9ba1d778ac18c7dfcf256e66328ec55ef08c1fb0e10ad41a81760154e86f986240082055a2c024b3363
7
+ data.tar.gz: 03ea21bf0625a57e368595faf8bbdf8a5bfed798a4663e43e92da26f89e793b897b1a175824f61a355e84158a3abde04bae7fba10a35bbbecdc8772345e757a3
data/README.md CHANGED
@@ -1,19 +1,20 @@
1
1
  # in-parallel
2
2
  A lightweight Ruby library with very simple syntax, making use of process.fork for parallelization
3
3
 
4
- Other popular Ruby librarys that do parallel execution support one primary use case - crunching through a large queue of small tasks as quickly and efficiently as possible. This library primarily supports the use case of executing a few larger tasks in parallel and managing the stdout and return values to make it easy to understand which processes are logging what, and what the outcome of the execution was. This library was created to be used by Puppet's Beaker test framework to enable parallel execution of some of the framework's tasks, and allow people within thier tests to execute code in parallel when wanted. This solution does not check to see how many processors you have, it just forks as many processes as you ask for. That means that it will handle a handful of parallel processes well, but could definitely overload your system with ruby processes if you try to spin up a LOT of processes. If you're looking for something simple and light-weight and on either linux or mac (forking processes is not supported on Windows), then this solution could be what you want.
4
+ Other popular Ruby libraries that do parallel execution support one primary use case - crunching through a large queue of small tasks as quickly and efficiently as possible. This library primarily supports the use case of executing a few larger tasks in parallel and managing the stdout and return values to make it easy to understand which processes are logging what, and what the outcome of the execution was. This library was created to be used by Puppet's Beaker test framework to enable parallel execution of some of the framework's tasks, and allow people within thier tests to execute code in parallel when wanted. This solution does not check to see how many processors you have, it just forks as many processes as you ask for. That means that it will handle a handful of parallel processes well, but could definitely overload your system with ruby processes if you try to spin up a LOT of processes. If you're looking for something simple and light-weight and on either linux or mac (forking processes is not supported on Windows), then this solution could be what you want.
5
5
 
6
6
  If you are looking for something to support executing a lot of tasks in parallel as efficiently as possible, you should take a look at the [parallel](https://github.com/grosser/parallel) project.
7
7
 
8
8
  ## Methods:
9
9
 
10
- ### run_in_parallel(&block)
10
+ ### run_in_parallel(timeout=nil, kill_all_on_error = false, &block)
11
11
  1. You can put whatever methods you want to execute in parallel into a block, and each method will be executed in parallel (unless the method is defined in kernel).
12
- 1. Any methods further down the stack won't be affected, only the ones directly within the block.
12
+ 1. Any methods further down the stack won't be affected, only the ones directly within the block.
13
13
  2. You can assign the results to instance variables and it just works, no dealing with an array or map of results.
14
14
  3. Log STDOUT and STDERR chunked per process to the console so that it is easy to see what happened in which process.
15
15
  4. Waits for each process in realtime and logs immediately upon completion of each process
16
- 5. If an exception is raised by a child process, it will immediately be re-raised in the primary process and kill all other still running child processes
16
+ 5. If an exception is raised by a child process, it will optionally (kill_all_on_error) be re-raised in the primary process and kill all other still running child processes. The default will wait for all processes to complete execution before re-raising any unhandled exception from the child processes.
17
+ 6. Times out by default at 30 minutes. Timeout default can be changed with InParallel::InParallelExecutor.parallel_default_timeout=X, or you can set the timeout param when calling the method
17
18
 
18
19
  ```ruby
19
20
  def method_with_param(name)
@@ -55,6 +56,38 @@ hello world
55
56
  hello world, bar
56
57
  ```
57
58
 
59
+ ### Enumerable.each_in_parallel(identifier=nil, timeout=(InParallel::InParallelExecutor.timeout), kill_all_on_error = false, &block)
60
+ 1. This is very similar to other solutions, except that it directly extends the Enumerable class with an each_in_parallel method, giving you the ability to pretty simply spawn a process for any item in an array or map.
61
+ 2. Identifies the block location (or caller location if the block does not have a source_location) in the console log to make it clear which block is being executed
62
+ 3. identifier param is only for logging, otherwise it will use the block source location.
63
+ 4. If an exception is raised by a child process, it will optionally (kill_all_on_error) be re-raised in the primary process and kill all other still running child processes. The default will wait for all processes to complete execution before re-raising any unhandled exception from the child processes.
64
+ 5. Times out by default at 30 minutes. Timeout default can be changed with InParallel::InParallelExecutor.parallel_default_timeout=X, or you can set the timeout param when calling the method
65
+
66
+ ```ruby
67
+ ["foo", "bar", "baz"].each_in_parallel { |item|
68
+ puts item
69
+ }
70
+
71
+ ```
72
+ STDOUT:
73
+ ```
74
+ 'each_in_parallel' spawned process for '/Users/samwoods/parallel_test/test.rb:77:in `block (2 levels) in <top (required)>'' - PID = '51600'
75
+ 'each_in_parallel' spawned process for '/Users/samwoods/parallel_test/test.rb:77:in `block (2 levels) in <top (required)>'' - PID = '51601'
76
+ 'each_in_parallel' spawned process for '/Users/samwoods/parallel_test/test.rb:77:in `block (2 levels) in <top (required)>'' - PID = '51602'
77
+
78
+ ------ Begin output for /Users/samwoods/parallel_test/test.rb:77:in `block (2 levels) in <top (required)>' - 51600
79
+ foo
80
+ ------ Completed output for /Users/samwoods/parallel_test/test.rb:77:in `block (2 levels) in <top (required)>' - 51600
81
+
82
+ ------ Begin output for /Users/samwoods/parallel_test/test.rb:77:in `block (2 levels) in <top (required)>' - 51601
83
+ bar
84
+ ------ Completed output for /Users/samwoods/parallel_test/test.rb:77:in `block (2 levels) in <top (required)>' - 51601
85
+
86
+ ------ Begin output for /Users/samwoods/parallel_test/test.rb:77:in `block (2 levels) in <top (required)>' - 51602
87
+ baz
88
+ ------ Completed output for /Users/samwoods/parallel_test/test.rb:77:in `block (2 levels) in <top (required)>' - 51602
89
+ ```
90
+
58
91
  ### run_in_background(ignore_results = true, &block)
59
92
  1. This does basically the same thing as run_in_parallel, except it does not wait for execution of all processes to complete, it returns immediately.
60
93
  2. You can optionally ignore results completely (default) or delay evaluating the results until later
@@ -101,31 +134,7 @@ hello world, bar
101
134
 
102
135
  ```
103
136
 
104
- ### Enumerable.each_in_parallel(&block)
105
- 1. This is very similar to other solutions, except that it directly extends the Enumerable class with an each_in_parallel method, giving you the ability to pretty simply spawn a process for any item in an array or map.
106
- 2. Identifies the block location (or caller location if the block does not have a source_location) in the console log to make it clear which block is being executed
107
-
108
- ```ruby
109
- ["foo", "bar", "baz"].each_in_parallel { |item|
110
- puts |item|
111
- }
112
-
113
- ```
114
- STDOUT:
115
- ```
116
- 'each_in_parallel' spawned process for '/Users/samwoods/parallel_test/test.rb:77:in `block (2 levels) in <top (required)>'' - PID = '51600'
117
- 'each_in_parallel' spawned process for '/Users/samwoods/parallel_test/test.rb:77:in `block (2 levels) in <top (required)>'' - PID = '51601'
118
- 'each_in_parallel' spawned process for '/Users/samwoods/parallel_test/test.rb:77:in `block (2 levels) in <top (required)>'' - PID = '51602'
119
-
120
- ------ Begin output for /Users/samwoods/parallel_test/test.rb:77:in `block (2 levels) in <top (required)>' - 51600
121
- foo
122
- ------ Completed output for /Users/samwoods/parallel_test/test.rb:77:in `block (2 levels) in <top (required)>' - 51600
123
-
124
- ------ Begin output for /Users/samwoods/parallel_test/test.rb:77:in `block (2 levels) in <top (required)>' - 51601
125
- bar
126
- ------ Completed output for /Users/samwoods/parallel_test/test.rb:77:in `block (2 levels) in <top (required)>' - 51601
127
-
128
- ------ Begin output for /Users/samwoods/parallel_test/test.rb:77:in `block (2 levels) in <top (required)>' - 51602
129
- baz
130
- ------ Completed output for /Users/samwoods/parallel_test/test.rb:77:in `block (2 levels) in <top (required)>' - 51602
131
- ```
137
+ ### wait_for_processes(timeout=nil, kill_all_on_error = false)
138
+ 1. Used only after run_in_background with ignore_results=false
139
+ 2. Optional args for timeout and kill_all_on_error
140
+ 3. See run_in_background for examples
data/in_parallel/version.rb CHANGED
@@ -1,3 +1,3 @@
1
1
  module InParallel
2
- VERSION = Version = '0.1.9'
2
+ VERSION = Version = '0.1.10'
3
3
  end
data/lib/in_parallel.rb CHANGED
@@ -1,11 +1,17 @@
1
+ require_relative 'parallel_logger'
1
2
  require_relative 'parallel_enumerable'
3
+ require 'tempfile'
4
+
2
5
  module InParallel
6
+ include ParallelLogger
7
+
3
8
  class InParallelExecutor
4
9
  # How many seconds between outputting to stdout that we are waiting for child processes.
5
10
  # 0 or < 0 means no signaling.
6
- @@signal_interval = 30
7
- @@timeout = 1800
8
- @@process_infos = []
11
+ @@parallel_signal_interval = 30
12
+ @@parallel_default_timeout = 1800
13
+
14
+ @@process_infos = []
9
15
  def self.process_infos
10
16
  @@process_infos
11
17
  end
@@ -21,44 +27,56 @@ module InParallel
21
27
  @@main_pid
22
28
  end
23
29
 
24
- def self.timeout
25
- @@timeout
30
+ def self.parallel_default_timeout
31
+ @@parallel_default_timeout
32
+ end
33
+
34
+ def self.parallel_default_timeout=(value)
35
+ @@parallel_default_timeout = value
36
+ end
37
+
38
+ def self.logger
39
+ @@logger
26
40
  end
27
41
 
28
- def self.timeout=(value)
29
- @@timeout = value
42
+ def self.logger=(value)
43
+ @@logger = value
30
44
  end
31
45
 
46
+ # Runs all methods within the block in parallel and waits for them to complete
47
+ #
32
48
  # Example - will spawn 2 processes, (1 for each method) wait until they both complete, and log STDOUT:
33
- # InParallel.run_in_parallel {
34
- # @result_1 = method1
35
- # @result_2 = method2
36
- # }
49
+ # InParallel.run_in_parallel {
50
+ # @result_1 = method1
51
+ # @result_2 = method2
52
+ # }
37
53
  # NOTE: Only supports assigning instance variables within the block, not local variables
38
- def self.run_in_parallel(timeout = @@timeout, kill_all_on_error = false, &block)
39
- if Process.respond_to?(:fork)
54
+ def self.run_in_parallel(timeout = @@parallel_default_timeout, kill_all_on_error = false, &block)
55
+ if fork_supported?
40
56
  proxy = BlankBindingParallelProxy.new(block.binding)
41
57
  proxy.instance_eval(&block)
42
58
  return wait_for_processes(proxy, block.binding, timeout, kill_all_on_error)
43
59
  end
44
- puts 'Warning: Fork is not supported on this OS, executing block normally'
60
+ # if fork is not supported
45
61
  block.call
46
62
  end
47
63
 
64
+ # Runs all methods within the block in parallel in the background
65
+ #
48
66
  # Example - Will spawn a process in the background to run puppet agent on two agents and return immediately:
49
- # Parallel.run_in_background {
50
- # @result_1 = method1
51
- # @result_2 = method2
52
- # }
53
- # # Do something else here before waiting for the process to complete
67
+ # Parallel.run_in_background {
68
+ # @result_1 = method1
69
+ # @result_2 = method2
70
+ # }
71
+ # # Do something else here before waiting for the process to complete
54
72
  #
55
- # # Optionally wait for the processes to complete before continuing.
56
- # # Otherwise use run_in_background(true) to clean up the process status and output immediately.
57
- # wait_for_processes(self)
58
- # NOTE: must call get_background_results to allow instance variables in calling object to be set,
59
- # otherwise @result_1 will evaluate to "unresolved_parallel_result_0"
73
+ # # Optionally wait for the processes to complete before continuing.
74
+ # # Otherwise use run_in_background(true) to clean up the process status and output immediately.
75
+ # wait_for_processes(self)
76
+ #
77
+ # NOTE: must call get_background_results to allow instance variables in calling object to be set, otherwise @result_1 will evaluate to "unresolved_parallel_result_0"
60
78
  def self.run_in_background(ignore_result = true, &block)
61
- if Process.respond_to?(:fork)
79
+ if fork_supported?
62
80
  proxy = BlankBindingParallelProxy.new(block.binding)
63
81
  proxy.instance_eval(&block)
64
82
 
@@ -71,25 +89,22 @@ module InParallel
71
89
  end
72
90
  return
73
91
  end
74
- puts 'Warning: Fork is not supported on this OS, executing block normally'
92
+ # if fork is not supported
75
93
  result = block.call
76
94
  return nil if ignore_result
77
95
  result
78
96
  end
79
97
 
80
- # Waits for all processes to complete and logs STDOUT and STDERR in chunks from any processes
81
- # that were triggered from this Parallel class
82
- # @param [Object] proxy - The instance of the proxy class that the method was executed within
83
- # (probably only useful when called by run_in_background)
84
- # @param [Object] binding - The binding of the block to assign return values to instance variables
85
- # (probably only useful when called by run_in_background)
98
+ # Waits for all processes to complete and logs STDOUT and STDERR in chunks from any processes that were triggered from this Parallel class
99
+ # @param [Object] proxy - The instance of the proxy class that the method was executed within (probably only useful when called by run_in_background)
100
+ # @param [Object] binding - The binding of the block to assign return values to instance variables (probably only useful when called by run_in_background)
86
101
  # @param [Int] timeout Time in seconds to wait before giving up on a child process
87
- # @param [Boolean] kill_all_on_error Whether to wait for all processes to complete, or fail immediately -
88
- # killing all other forked processes - when one process errors.
102
+ # @param [Boolean] kill_all_on_error Whether to wait for all processes to complete, or fail immediately - killing all other forked processes - when one process errors.
89
103
  def self.wait_for_processes(proxy = self, binding = nil, timeout = nil, kill_all_on_error = false)
90
104
  raise_error = nil
91
- timeout ||= @@timeout
105
+ timeout ||= @@parallel_default_timeout
92
106
  trap(:INT) do
107
+ # Can't use logger inside of trap
93
108
  puts "Warning, recieved interrupt. Processing child results and exiting."
94
109
  kill_child_processes
95
110
  end
@@ -100,8 +115,8 @@ module InParallel
100
115
  start_time = Time.now
101
116
  timer = start_time
102
117
  while !@@process_infos.empty? do
103
- if @@signal_interval > 0 && Time.now > timer + @@signal_interval
104
- puts 'Waiting for child processes.'
118
+ if @@parallel_signal_interval > 0 && Time.now > timer + @@parallel_signal_interval
119
+ @@logger.debug 'Waiting for child processes.'
105
120
  timer = Time.now
106
121
  end
107
122
  if Time.now > start_time + timeout
@@ -116,21 +131,23 @@ module InParallel
116
131
  # the process completed, get the result and rethrow on error.
117
132
  begin
118
133
  # Print the STDOUT and STDERR for each process with signals for start and end
119
- puts "\n------ Begin output for #{process_info[:method_sym]} - #{process_info[:pid]}\n"
120
- puts File.new(process_info[:std_out], 'r').readlines
121
- puts "------ Completed output for #{process_info[:method_sym]} - #{process_info[:pid]}\n"
134
+ @@logger.info "------ Begin output for #{process_info[:method_sym]} - #{process_info[:pid]}"
135
+ # Content from the other thread will already be pre-pended with log stuff (info, warn, date/time, etc)
136
+ # So don't use logger, just use puts.
137
+ puts " " + File.new(process_info[:std_out], 'r').readlines.join(" ")
138
+ @@logger.info "------ Completed output for #{process_info[:method_sym]} - #{process_info[:pid]}"
122
139
  result = process_info[:result].read
123
140
  marshalled_result = (result.nil? || result.empty?) ? result : Marshal.load(result)
141
+ # Kill all other processes and let them log their stdout before re-raising
142
+ # if a child process raised an error.
124
143
  if marshalled_result.is_a?(Exception)
125
144
  raise_error = marshalled_result.dup
126
145
  kill_child_processes if kill_all_on_error
127
146
  marshalled_result = nil
128
147
  end
129
148
  results_map[process_info[:index]] = {process_info[:tmp_result] => marshalled_result}
130
- File.delete(process_info[:std_out])
131
- # Kill all other processes and let them log their stdout before re-raising
132
- # if a child process raised an error.
133
149
  ensure
150
+ File.delete(process_info[:std_out]) if File.exists?(process_info[:std_out])
134
151
  # close the read end pipe
135
152
  process_info[:result].close unless process_info[:result].closed?
136
153
  @@process_infos.delete(process_info)
@@ -164,21 +181,22 @@ module InParallel
164
181
  ret_val = nil
165
182
  # Communicate the return value of the method or block
166
183
  read_result, write_result = IO.pipe
167
- Dir.mkdir('tmp') unless Dir.exists? 'tmp'
168
184
  pid = fork do
185
+ Dir.mkdir('tmp') unless Dir.exists? 'tmp'
186
+ stdout_file = File.new("tmp/pp_#{Process.pid}", 'w')
169
187
  exit_status = 0
170
188
  trap(:INT) do
171
- puts("Warning: Interrupt received in child process; exiting #{Process.pid}")
189
+ # Can't use logger inside of trap
190
+ puts "Warning: Interrupt received in child process; exiting #{Process.pid}"
172
191
  kill_child_processes
173
192
  return
174
193
  end
175
- write_file = File.new("tmp/parallel_process_#{Process.pid}", 'w')
176
194
 
177
195
  # IO buffer is 64kb, which isn't much... if debug logging is turned on,
178
196
  # this can be exceeded before a process completes.
179
197
  # Storing output in file rather than using IO.pipe
180
- STDOUT.reopen(write_file)
181
- STDERR.reopen(write_file)
198
+ STDOUT.reopen(stdout_file)
199
+ STDERR.reopen(stdout_file)
182
200
 
183
201
  begin
184
202
  # close subprocess's copy of read_result since it only needs to write
@@ -191,7 +209,7 @@ module InParallel
191
209
  begin
192
210
  ret_val = ret_val.dup
193
211
  rescue StandardError => err
194
- puts "Warning: return value from child process #{ret_val} " +
212
+ @@logger.warn "Warning: return value from child process #{ret_val} " +
195
213
  "could not be transferred to parent process: #{err.message}"
196
214
  end
197
215
  end
@@ -199,11 +217,11 @@ module InParallel
199
217
  begin
200
218
  Marshal.dump(ret_val, write_result) unless ret_val.nil?
201
219
  rescue StandardError => err
202
- puts "Warning: return value from child process #{ret_val} " +
220
+ @@logger.warn "Warning: return value from child process #{ret_val} " +
203
221
  "could not be transferred to parent process: #{err.message}"
204
222
  end
205
223
  rescue Exception => err
206
- puts "Error in process #{pid}: #{err.message}"
224
+ @@logger.error "Error in process #{pid}: #{err.message}"
207
225
  # Return the error if an error is rescued so we can re-throw in the main process.
208
226
  Marshal.dump(err, write_result)
209
227
  exit_status = 1
@@ -212,6 +230,8 @@ module InParallel
212
230
  exit exit_status
213
231
  end
214
232
  end
233
+
234
+ @@logger.info "Forked process for #{method_sym} - PID = '#{pid}'"
215
235
  write_result.close
216
236
  # Process.detach returns a thread that will be nil if the process is still running and thr if not.
217
237
  # This allows us to check to see if processes have exited without having to call the blocking Process.wait functions.
@@ -220,7 +240,7 @@ module InParallel
220
240
  process_info = { :wait_thread => wait_thread,
221
241
  :pid => pid,
222
242
  :method_sym => method_sym,
223
- :std_out => "tmp/parallel_process_#{pid}",
243
+ :std_out => "tmp/pp_#{pid}",
224
244
  :result => read_result,
225
245
  :tmp_result => "unresolved_parallel_result_#{@@result_id}",
226
246
  :index => @@process_infos.count }
@@ -229,6 +249,12 @@ module InParallel
229
249
  process_info
230
250
  end
231
251
 
252
+ def self.fork_supported?
253
+ @@supported ||= Process.respond_to?(:fork)
254
+ @@logger.warn 'Warning: Fork is not supported on this OS, executing block normally' unless @@supported
255
+ @@supported
256
+ end
257
+
232
258
  def self.kill_child_processes
233
259
  @@process_infos.each { |process_info|
234
260
  # Send INT to each child process so it returns and can print stdout and stderr to console before exiting.
@@ -276,49 +302,65 @@ module InParallel
276
302
  def method_missing(method_sym, *args, &block)
277
303
  if InParallelExecutor.main_pid == ::Process.pid
278
304
  out = InParallelExecutor._execute_in_parallel("'#{method_sym.to_s}' #{caller_locations[0].to_s}", @object.eval('self')) {send(method_sym, *args, &block)}
279
- puts "Forked process for '#{method_sym}' - PID = '#{out[:pid]}'\n"
280
305
  out[:tmp_result]
281
306
  end
282
307
  end
283
308
  end
284
309
  end
285
310
 
286
- # Executes each method within a block in a different process
311
+ InParallelExecutor.logger = @logger
312
+
313
+ def parallel_signal_interval
314
+ InParallelExecutor.parallel_signal_interval
315
+ end
316
+
317
+ def parallel_signal_interval=(value)
318
+ InParallelExecutor.parallel_signal_interval = value
319
+ end
320
+
321
+ def parallel_default_timeout
322
+ InParallelExecutor.parallel_default_timeout
323
+ end
324
+
325
+ def parallel_default_timeout=(value)
326
+ InParallelExecutor.parallel_default_timeout = value
327
+ end
328
+
329
+ # Executes each method within a block in a different process.
330
+ #
287
331
  # Example - Will spawn a process in the background to execute each method
288
- # Parallel.run_in_parallel {
289
- # @result_1 = method1
290
- # @result_2 = method2
291
- # }
292
- # NOTE - Only instance variables can be assigned the return values of the methods within the block.
293
- # Local variables will not be assigned any values.
332
+ # Parallel.run_in_parallel {
333
+ # @result_1 = method1
334
+ # @result_2 = method2
335
+ # }
336
+ # NOTE - Only instance variables can be assigned the return values of the methods within the block. Local variables will not be assigned any values.
294
337
  # @param [Int] timeout Time in seconds to wait before giving up on a child process
295
- # @param [Boolean] kill_all_on_error Whether to wait for all processes to complete, or fail immediately -
296
- # killing all other forked processes - when one process errors.
338
+ # @param [Boolean] kill_all_on_error Whether to wait for all processes to complete, or fail immediately - killing all other forked processes - when one process errors.
297
339
  # @param [Block] block This method will yield to a block of code passed by the caller
298
340
  # @return [Array<Result>, Result] the return values of each method within the block
299
341
  def run_in_parallel(timeout=nil, kill_all_on_error = false, &block)
300
- timeout ||= InParallelExecutor.timeout
342
+ timeout ||= InParallelExecutor.parallel_default_timeout
301
343
  InParallelExecutor.run_in_parallel(timeout, kill_all_on_error, &block)
302
344
  end
303
345
 
304
- # Forks a process for each method within a block and returns immediately
346
+ # Forks a process for each method within a block and returns immediately.
347
+ #
305
348
  # Example 1 - Will fork a process in the background to execute each method and return immediately:
306
- # Parallel.run_in_background {
307
- # @result_1 = method1
308
- # @result_2 = method2
309
- # }
349
+ # Parallel.run_in_background {
350
+ # @result_1 = method1
351
+ # @result_2 = method2
352
+ # }
310
353
  #
311
354
  # Example 2 - Will fork a process in the background to execute each method, return immediately, then later
312
355
  # wait for the process to complete, printing it's STDOUT and assigning return values to instance variables:
313
- # Parallel.run_in_background(false) {
314
- # @result_1 = method1
315
- # @result_2 = method2
316
- # }
317
- # # Do something else here before waiting for the process to complete
356
+ # Parallel.run_in_background(false) {
357
+ # @result_1 = method1
358
+ # @result_2 = method2
359
+ # }
360
+ # # Do something else here before waiting for the process to complete
318
361
  #
319
- # wait_for_processes
320
- # NOTE: must call wait_for_processes to allow instance variables within the block to be set,
321
- # otherwise results will evaluate to "unresolved_parallel_result_X"
362
+ # wait_for_processes
363
+ # NOTE: must call wait_for_processes to allow instance variables within the block to be set, otherwise results will evaluate to "unresolved_parallel_result_X"
322
364
  # @param [Boolean] ignore_result True if you do not care about the STDOUT or return value of the methods executing in the background
323
365
  # @param [Block] block This method will yield to a block of code passed by the caller
324
366
  # @return [Array<Result>, Result] the return values of each method within the block
@@ -326,14 +368,12 @@ module InParallel
326
368
  InParallelExecutor.run_in_background(ignore_result, &block)
327
369
  end
328
370
 
329
- # Waits for all processes started by run_in_background to complete execution, then prints STDOUT
330
- # and assigns return values to instance variables. See :run_in_background
371
+ # Waits for all processes started by run_in_background to complete execution, then prints STDOUT and assigns return values to instance variables. See :run_in_background
331
372
  # @param [Int] timeout Time in seconds to wait before giving up on a child process
332
- # @param [Boolean] kill_all_on_error Whether to wait for all processes to complete, or fail immediately -
333
- # killing all other forked processes - when one process errors.
373
+ # @param [Boolean] kill_all_on_error Whether to wait for all processes to complete, or fail immediately - killing all other forked processes - when one process errors.
334
374
  # @return [Array<Result>, Result] the temporary return values of each method within the block
335
375
  def wait_for_processes(timeout=nil, kill_all_on_error = false)
336
- timeout ||= InParallelExecutor.timeout
376
+ timeout ||= InParallelExecutor.parallel_default_timeout
337
377
  InParallelExecutor.wait_for_processes(nil, nil, timeout, kill_all_on_error)
338
378
  end
339
379
  end
data/lib/parallel_enumerable.rb CHANGED
@@ -1,26 +1,25 @@
1
1
  # Extending Enumerable to make it easy to do any .each in parallel
2
2
  module Enumerable
3
3
  # Executes each iteration of the block in parallel
4
- # Example - Will execute each iteration in a separate process, in parallel,
5
- # log STDOUT per process, and return an array of results.
6
- # my_array = [1,2,3]
7
- # my_array.each_in_parallel { |int|
8
- # my_method(int)
9
- # }
4
+ #
5
+ # Example - Will execute each iteration in a separate process, in parallel, log STDOUT per process, and return an array of results.
6
+ # my_array = [1,2,3]
7
+ # my_array.each_in_parallel { |int|
8
+ # my_method(int)
9
+ # }
10
10
  # @param [String] identifier - Optional identifier for logging purposes only. Will use the block location by default.
11
11
  # @param [Int] timeout - Seconds to wait for a forked process to complete before timing out
12
12
  # @return [Array<Object>] results - the return value of each block execution.
13
- def each_in_parallel(identifier=nil, timeout=(InParallel::InParallelExecutor.timeout), kill_all_on_error = false, &block)
14
- if Process.respond_to?(:fork) && count > 1
13
+ def each_in_parallel(identifier=nil, timeout=(InParallel::InParallelExecutor.parallel_default_timeout), kill_all_on_error = false, &block)
14
+ if InParallel::InParallelExecutor.fork_supported? && count > 1
15
15
  identifier ||= "#{caller_locations[0]}"
16
16
  each do |item|
17
17
  out = InParallel::InParallelExecutor._execute_in_parallel(identifier) {block.call(item)}
18
- puts "'each_in_parallel' forked process for '#{identifier}' - PID = '#{out[:pid]}'\n"
19
18
  end
20
19
  # return the array of values, no need to look up from the map.
21
20
  return InParallel::InParallelExecutor.wait_for_processes(nil, block.binding, timeout, kill_all_on_error)
22
21
  end
23
- puts 'Warning: Fork is not supported on this OS, executing block normally' unless Process.respond_to? :fork
22
+ # If fork is not supported
24
23
  block.call
25
24
  each(&block)
26
25
  end
data/lib/parallel_logger.rb ADDED
@@ -0,0 +1,13 @@
1
+ require 'logger'
2
+ module InParallel
3
+ module ParallelLogger
4
+ def self.included(base)
5
+ # Use existing logger if it is defined
6
+ unless(base.instance_variables.include?(:@logger) && base.logger)
7
+ logger = Logger.new(STDOUT)
8
+ logger.send(:extend, self)
9
+ base.instance_variable_set(:@logger, logger)
10
+ end
11
+ end
12
+ end
13
+ end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: in-parallel
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.9
4
+ version: 0.1.10
5
5
  platform: ruby
6
6
  authors:
7
7
  - samwoods1
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2016-06-21 00:00:00.000000000 Z
11
+ date: 2016-06-27 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
@@ -66,6 +66,7 @@ files:
66
66
  - in_parallel/version.rb
67
67
  - lib/in_parallel.rb
68
68
  - lib/parallel_enumerable.rb
69
+ - lib/parallel_logger.rb
69
70
  homepage: https://github.com/samwoods1/in-parallel
70
71
  licenses:
71
72
  - MIT