in-parallel 0.1.10 → 0.1.11

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 616a0cfcbc49b019d69c306e9e919a86efdd9fee
-  data.tar.gz: 4e7aba20c6ca158d124242c74ce44ca59fbf9179
+  metadata.gz: ecbae8756fd4310b2b2330b342f7ada3ee48dae6
+  data.tar.gz: c42d6e4ffb5d87d1418d9bccbc65c4ec3347f5b5
 SHA512:
-  metadata.gz: d9e961955959052dd40268155c142deefa2950145519d9ba1d778ac18c7dfcf256e66328ec55ef08c1fb0e10ad41a81760154e86f986240082055a2c024b3363
-  data.tar.gz: 03ea21bf0625a57e368595faf8bbdf8a5bfed798a4663e43e92da26f89e793b897b1a175824f61a355e84158a3abde04bae7fba10a35bbbecdc8772345e757a3
+  metadata.gz: 0510bb42cb5342dee946b5f383acd9da2c0277bc60c466c80f17ab10b7f243d34672868c2f95c36525781a6fe95c7d47e52bb4003c6d64d67886201b832aaf3c
+  data.tar.gz: 30d1528fa0ce04a39636f10c0e3e969315e5c679ada2b763f8b0046a802ee7a15c5d1bd6abcb61aa8a60b63a2e39cea44c706b60cb17d36f40e03f2d635d10f0

data/README.md

@@ -1,16 +1,16 @@
 # in-parallel
-A lightweight Ruby library with very simple syntax, making use of process.fork for parallelization
+A lightweight Ruby library with very simple syntax, making use of Process.fork to execute code in parallel.
 
-Other popular Ruby libraries that do parallel execution support one primary use case - crunching through a large queue of small tasks as quickly and efficiently as possible.  This library primarily supports the use case of executing a few larger tasks in parallel and managing the stdout and return values to make it easy to understand which processes are logging what, and what the outcome of the execution was. This library was created to be used by Puppet's Beaker test framework to enable parallel execution of some of the framework's tasks, and allow people within thier tests to execute code in parallel when wanted.  This solution does not check to see how many processors you have, it just forks as many processes as you ask for.  That means that it will handle a handful of parallel processes well, but could definitely overload your system with ruby processes if you try to spin up a LOT of processes.  If you're looking for something simple and light-weight and on either linux or mac (forking processes is not supported on Windows), then this solution could be what you want.
+Other popular Ruby libraries that do parallel execution support one primary use case - crunching through a large queue of small tasks as quickly and efficiently as possible.  This library primarily supports the use case of executing a few larger tasks in parallel and managing the stdout and return values to make it easy to understand which processes are logging what, and what the outcome of the execution was. This library was created to be used by Puppet's Beaker test framework to enable parallel execution of some of the framework's tasks, and allow users to execute code in parallel within their tests.  This solution does not check to see how many processors you have, it just forks as many processes as you ask for.  That means that it will handle a handful of parallel processes well, but could definitely overload your system with ruby processes if you try to spin up a LOT of processes.  If you're looking for something intuitive and simple, but with a slight memory and processing overhead, and are on either Linux or Mac (forking processes is not supported on Windows), then this solution is  what you want.
 
-If you are looking for something to support executing a lot of tasks in parallel as efficiently as possible, you should take a look at the [parallel](https://github.com/grosser/parallel) project.
+If you are looking for something that excels at executing a large queue of tasks in parallel as efficiently as possible, you should take a look at the [parallel](https://github.com/grosser/parallel) project.
 
 ## Methods:
 
 ### run_in_parallel(timeout=nil, kill_all_on_error = false, &block)
-1. You can put whatever methods you want to execute in parallel into a block, and each method will be executed in parallel (unless the method is defined in kernel). 
+1. Each method in a block will be executed in parallel (unless the method is defined in Kernel or BaseObject).
     1. Any methods further down the stack won't be affected, only the ones directly within the block.
-2. You can assign the results to instance variables and it just works, no dealing with an array or map of results.
+2. You can assign return values to instance variables and it 'just works'.
 3. Log STDOUT and STDERR chunked per process to the console so that it is easy to see what happened in which process.
 4. Waits for each process in realtime and logs immediately upon completion of each process
 5. If an exception is raised by a child process, it will optionally (kill_all_on_error) be re-raised in the primary process and kill all other still running child processes. The default will wait for all processes to complete execution before re-raising any unhandled exception from the child processes.
@@ -33,10 +33,10 @@ If you are looking for something to support executing a lot of tasks in parallel
   # Example:
   # will spawn 2 processes, (1 for each method) wait until they both complete, log chunked STDOUT/STDERR for
   # each process and assign the method return values to instance variables:
-  InParallel.run_in_parallel {
+  run_in_parallel do
     @result_1 = method_with_param('world')
     @result_2 = method_without_param
-  }
+  end
   
   puts "#{@result_1}, #{@result_2[:foo]}"
 ```
@@ -64,9 +64,7 @@ hello world, bar
 5. Times out by default at 30 minutes. Timeout default can be changed with InParallel::InParallelExecutor.parallel_default_timeout=X, or you can set the timeout param when calling the method
 
 ```ruby
-  ["foo", "bar", "baz"].each_in_parallel { |item|
-    puts item
-  }
+  ["foo", "bar", "baz"].each_in_parallel { |item| puts item }
 
 ```
 STDOUT:
@@ -98,14 +96,12 @@ baz
   
   def create_file_with_delay(file_path)
     sleep 2
-    File.open(file_path, 'w') { |f| f.write('contents')}
+    File.open(file_path, 'w') { |f| f.write('contents') }
     return true
   end
   
   # Example 1 - ignore results
-  run_in_background{
-    create_file_with_delay(TMP_FILE)
-  }
+  run_in_background { create_file_with_delay(TMP_FILE) }
   
   # Should not exist immediately upon block completion
   puts(File.exists?(TMP_FILE)) # false
@@ -114,15 +110,11 @@ baz
   puts(File.exists?(TMP_FILE)) # true
   
   # Example 2 - delay results
-  run_in_background(false){
-    @result = create_file_with_delay(TMP_FILE)
-  }
+  run_in_background(false) { @result = create_file_with_delay(TMP_FILE) }
   
   # Do something else
   
-  run_in_background(false){
-    @result2 = create_file_with_delay('/tmp/someotherfile.txt')
-  }
+  run_in_background(false) { @result2 = create_file_with_delay('/tmp/someotherfile.txt') }
   
   # @result has not been assigned yet
   puts @result >> "unresolved_parallel_result_0"

data/in_parallel/version.rb

@@ -1,3 +1,3 @@
 module InParallel
-  VERSION = Version = '0.1.10'
+  VERSION = Version = '0.1.11'
 end

data/lib/in_parallel.rb

@@ -11,13 +11,14 @@ module InParallel
     @@parallel_signal_interval = 30
     @@parallel_default_timeout = 1800
 
-    @@process_infos            = []
+    @@process_infos = []
+
     def self.process_infos
       @@process_infos
     end
 
     @@background_objs = []
-    @@result_id = 0
+    @@result_id       = 0
 
     @@pids = []
 
@@ -46,10 +47,10 @@ module InParallel
     # Runs all methods within the block in parallel and waits for them to complete
     #
     # Example - will spawn 2 processes, (1 for each method) wait until they both complete, and log STDOUT:
-    #   InParallel.run_in_parallel {
+    #   InParallel.run_in_parallel do
     #     @result_1 = method1
     #     @result_2 = method2
-    #   }
+    #   end
     # NOTE: Only supports assigning instance variables within the block, not local variables
     def self.run_in_parallel(timeout = @@parallel_default_timeout, kill_all_on_error = false, &block)
       if fork_supported?
@@ -64,10 +65,10 @@ module InParallel
     # Runs all methods within the block in parallel in the background
     #
     # Example - Will spawn a process in the background to run puppet agent on two agents and return immediately:
-    #   Parallel.run_in_background {
+    #   Parallel.run_in_background do
     #     @result_1 = method1
     #     @result_2 = method2
-    #   }
+    #   end
     #   # Do something else here before waiting for the process to complete
     #
     #   # Optionally wait for the processes to complete before continuing.
@@ -84,7 +85,7 @@ module InParallel
           Process.detach(@@process_infos.last[:pid])
           @@process_infos.pop
         else
-          @@background_objs << {:proxy => proxy, :target => block.binding}
+          @@background_objs << { :proxy => proxy, :target => block.binding }
           return process_infos.last[:tmp_result]
         end
         return
@@ -102,7 +103,7 @@ module InParallel
     # @param [Boolean] kill_all_on_error Whether to wait for all processes to complete, or fail immediately - killing all other forked processes - when one process errors.
     def self.wait_for_processes(proxy = self, binding = nil, timeout = nil, kill_all_on_error = false)
       raise_error = nil
-      timeout ||= @@parallel_default_timeout
+      timeout     ||= @@parallel_default_timeout
       trap(:INT) do
         # Can't use logger inside of trap
         puts "Warning, recieved interrupt.  Processing child results and exiting."
@@ -112,8 +113,8 @@ module InParallel
       # Custom process to wait so that we can do things like time out, and kill child processes if
       # one process returns with an error before the others complete.
       results_map = Array.new(@@process_infos.count)
-      start_time = Time.now
-      timer = start_time
+      start_time  = Time.now
+      timer       = start_time
       while !@@process_infos.empty? do
         if @@parallel_signal_interval > 0 && Time.now > timer + @@parallel_signal_interval
           @@logger.debug 'Waiting for child processes.'
@@ -123,7 +124,7 @@ module InParallel
           kill_child_processes
           raise_error = ::RuntimeError.new("Child process ran longer than timeout of #{timeout}")
         end
-        @@process_infos.each {|process_info|
+        @@process_infos.each do |process_info|
           # wait up to half a second for each thread to see if it is complete, if not, check the next thread.
           # returns immediately if the process has completed.
           thr = process_info[:wait_thread].join(0.5)
@@ -136,7 +137,7 @@ module InParallel
               # So don't use logger, just use puts.
               puts "  " + File.new(process_info[:std_out], 'r').readlines.join("  ")
               @@logger.info "------ Completed output for #{process_info[:method_sym]} - #{process_info[:pid]}"
-              result = process_info[:result].read
+              result            = process_info[:result].read
               marshalled_result = (result.nil? || result.empty?) ? result : Marshal.load(result)
               # Kill all other processes and let them log their stdout before re-raising
               # if a child process raised an error.
@@ -145,7 +146,7 @@ module InParallel
                 kill_child_processes if kill_all_on_error
                 marshalled_result = nil
               end
-              results_map[process_info[:index]] = {process_info[:tmp_result] => marshalled_result}
+              results_map[process_info[:index]] = { process_info[:tmp_result] => marshalled_result }
             ensure
               File.delete(process_info[:std_out]) if File.exists?(process_info[:std_out])
               # close the read end pipe
@@ -153,7 +154,7 @@ module InParallel
               @@process_infos.delete(process_info)
             end
           end
-        }
+        end
       end
 
       results = []
@@ -166,9 +167,7 @@ module InParallel
       # If there are background_objs AND results, don't return the background obj results
       # (which would mess up expected results from each_in_parallel),
       # but do process their results in case they are assigned to instance variables
-      @@background_objs.each {|obj|
-         result_lookup(obj[:proxy], obj[:target], results_map)
-      }
+      @@background_objs.each { |obj| result_lookup(obj[:proxy], obj[:target], results_map) }
       @@background_objs.clear
 
       raise raise_error unless raise_error.nil?
@@ -178,10 +177,10 @@ module InParallel
 
     # private method to execute some code in a separate process and store the STDOUT and STDERR for later retrieval
     def self._execute_in_parallel(method_sym, obj = self, &block)
-      ret_val = nil
+      ret_val                   = nil
       # Communicate the return value of the method or block
       read_result, write_result = IO.pipe
-      pid = fork do
+      pid                       = fork do
         Dir.mkdir('tmp') unless Dir.exists? 'tmp'
         stdout_file = File.new("tmp/pp_#{Process.pid}", 'w')
         exit_status = 0
@@ -204,13 +203,13 @@ module InParallel
           ret_val = obj.instance_eval(&block)
           # Write the result to the write_result IO stream.
           # Have to serialize the value so it can be transmitted via IO
-          if(!ret_val.nil? && ret_val.singleton_methods && ret_val.class != TrueClass && ret_val.class != FalseClass && ret_val.class != Fixnum)
+          if (!ret_val.nil? && ret_val.singleton_methods && ret_val.class != TrueClass && ret_val.class != FalseClass && ret_val.class != Fixnum)
             #in case there are other types that can't be duped
             begin
               ret_val = ret_val.dup
             rescue StandardError => err
               @@logger.warn "Warning: return value from child process #{ret_val} " +
-                       "could not be transferred to parent process: #{err.message}"
+                                "could not be transferred to parent process: #{err.message}"
             end
           end
           # In case there are other types that can't be dumped
@@ -218,7 +217,7 @@ module InParallel
             Marshal.dump(ret_val, write_result) unless ret_val.nil?
           rescue StandardError => err
             @@logger.warn "Warning: return value from child process #{ret_val} " +
-                     "could not be transferred to parent process: #{err.message}"
+                              "could not be transferred to parent process: #{err.message}"
           end
         rescue Exception => err
           @@logger.error "Error in process #{pid}: #{err.message}"
@@ -235,15 +234,15 @@ module InParallel
       write_result.close
       # Process.detach returns a thread that will be nil if the process is still running and thr if not.
       # This allows us to check to see if processes have exited without having to call the blocking Process.wait functions.
-      wait_thread = Process.detach(pid)
+      wait_thread  = Process.detach(pid)
       # store the IO object with the STDOUT and waiting thread for each pid
       process_info = { :wait_thread => wait_thread,
-                       :pid => pid,
-                       :method_sym => method_sym,
-                       :std_out => "tmp/pp_#{pid}",
-                       :result => read_result,
-                       :tmp_result => "unresolved_parallel_result_#{@@result_id}",
-                       :index => @@process_infos.count }
+                       :pid         => pid,
+                       :method_sym  => method_sym,
+                       :std_out     => "tmp/pp_#{pid}",
+                       :result      => read_result,
+                       :tmp_result  => "unresolved_parallel_result_#{@@result_id}",
+                       :index       => @@process_infos.count }
       @@process_infos.push(process_info)
       @@result_id += 1
       process_info
@@ -256,35 +255,37 @@ module InParallel
     end
 
     def self.kill_child_processes
-      @@process_infos.each { |process_info|
+      @@process_infos.each do |process_info|
         # Send INT to each child process so it returns and can print stdout and stderr to console before exiting.
         begin
           Process.kill("INT", process_info[:pid])
         rescue Errno::ESRCH
           # If one of the other processes has completed in the very short time before we try to kill it, handle the exception
         end
-      }
+      end
     end
+
     private_class_method :kill_child_processes
 
     # Private method to lookup results from the results_map and replace the
     # temp values with actual return values
     def self.result_lookup(proxy_obj, target_obj, results_map)
       target_obj = eval('self', target_obj)
-      proxy_obj ||= target_obj
-      vars = (proxy_obj.instance_variables)
-      results = []
-      results_map.each { |tmp_result|
+      proxy_obj  ||= target_obj
+      vars       = proxy_obj.instance_variables
+      results    = []
+      results_map.each do |tmp_result|
         results << tmp_result.values[0]
-        vars.each {|var|
+        vars.each do |var|
           if proxy_obj.instance_variable_get(var) == tmp_result.keys[0]
             target_obj.instance_variable_set(var, tmp_result.values[0])
             break
           end
-        }
-      }
+        end
+      end
       results
     end
+
     private_class_method :result_lookup
 
     # Proxy class used to wrap each method execution in a block and run it in parallel
@@ -294,14 +295,15 @@ module InParallel
       include ::Kernel
 
       def initialize(obj)
-        @object = obj
+        @object    = obj
         @result_id = 0
       end
 
       # All methods within the block should show up as missing (unless defined in :Kernel)
       def method_missing(method_sym, *args, &block)
         if InParallelExecutor.main_pid == ::Process.pid
-          out = InParallelExecutor._execute_in_parallel("'#{method_sym.to_s}' #{caller_locations[0].to_s}", @object.eval('self')) {send(method_sym, *args, &block)}
+          out = InParallelExecutor._execute_in_parallel("'#{method_sym.to_s}' #{caller_locations[0].to_s}",
+                                                        @object.eval('self')) { send(method_sym, *args, &block) }
           out[:tmp_result]
         end
       end
@@ -310,18 +312,24 @@ module InParallel
 
   InParallelExecutor.logger = @logger
 
+  # Gets how many seconds to wait between logging a 'Waiting for child processes.'
   def parallel_signal_interval
     InParallelExecutor.parallel_signal_interval
   end
 
+  # Sets how many seconds to wait between logging a 'Waiting for child processes.'
+  # @param [Int] value Time in seconds to wait before logging 'Waiting for child processes.'
   def parallel_signal_interval=(value)
     InParallelExecutor.parallel_signal_interval = value
   end
 
+  # Gets how many seconds to wait before timing out a forked child process and raising an exception
   def parallel_default_timeout
     InParallelExecutor.parallel_default_timeout
   end
 
+  # Sets how many seconds to wait before timing out a forked child process and raising an exception
+  # @param [Int] value Time in seconds to wait before timing out and raising an exception
   def parallel_default_timeout=(value)
     InParallelExecutor.parallel_default_timeout = value
   end
@@ -329,10 +337,10 @@ module InParallel
   # Executes each method within a block in a different process.
   #
   # Example - Will spawn a process in the background to execute each method
-  #   Parallel.run_in_parallel {
+  #   Parallel.run_in_parallel do
   #     @result_1 = method1
   #     @result_2 = method2
-  #   }
+  #   end
   # NOTE - Only instance variables can be assigned the return values of the methods within the block. Local variables will not be assigned any values.
   # @param [Int] timeout Time in seconds to wait before giving up on a child process
   # @param [Boolean] kill_all_on_error Whether to wait for all processes to complete, or fail immediately - killing all other forked processes - when one process errors.
@@ -346,17 +354,17 @@ module InParallel
   # Forks a process for each method within a block and returns immediately.
   #
   # Example 1 - Will fork a process in the background to execute each method and return immediately:
-  #   Parallel.run_in_background {
+  #   Parallel.run_in_background do
   #     @result_1 = method1
   #     @result_2 = method2
-  #   }
+  #   end
   #
   # Example 2 - Will fork a process in the background to execute each method, return immediately, then later
   # wait for the process to complete, printing it's STDOUT and assigning return values to instance variables:
-  #   Parallel.run_in_background(false) {
+  #   Parallel.run_in_background(false) do
   #     @result_1 = method1
   #     @result_2 = method2
-  #   }
+  #   end
   #   # Do something else here before waiting for the process to complete
   #
   #   wait_for_processes

data/lib/parallel_enumerable.rb

@@ -4,9 +4,7 @@ module Enumerable
   #
   # Example - Will execute each iteration in a separate process, in parallel, log STDOUT per process, and return an array of results.
   #   my_array = [1,2,3]
-  #   my_array.each_in_parallel { |int|
-  #     my_method(int)
-  #   }
+  #   my_array.each_in_parallel { |int| my_method(int) }
   # @param [String] identifier - Optional identifier for logging purposes only. Will use the block location by default.
   # @param [Int] timeout - Seconds to wait for a forked process to complete before timing out
   # @return [Array<Object>] results - the return value of each block execution.
@@ -14,7 +12,7 @@ module Enumerable
     if InParallel::InParallelExecutor.fork_supported? && count > 1
       identifier ||= "#{caller_locations[0]}"
       each do |item|
-        out = InParallel::InParallelExecutor._execute_in_parallel(identifier) {block.call(item)}
+        InParallel::InParallelExecutor._execute_in_parallel(identifier) { block.call(item) }
       end
       # return the array of values, no need to look up from the map.
       return InParallel::InParallelExecutor.wait_for_processes(nil, block.binding, timeout, kill_all_on_error)

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: in-parallel
 version: !ruby/object:Gem::Version
-  version: 0.1.10
+  version: 0.1.11
 platform: ruby
 authors:
 - samwoods1
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2016-06-27 00:00:00.000000000 Z
+date: 2016-06-29 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler