parallel-each 1.0.0

data/CHANGELOG

@@ -0,0 +1,5 @@
+v1.0.0
+=====
+
+ * Parallel iteration over item in an Enumerable
+

data/LICENSE

@@ -0,0 +1,165 @@
+		   GNU LESSER GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+  This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+  0. Additional Definitions.
+
+  As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+  "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+  An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+  A "Combined Work" is a work produced by combining or linking an
+Application with the Library.  The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+  The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+  The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+  1. Exception to Section 3 of the GNU GPL.
+
+  You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+  2. Conveying Modified Versions.
+
+  If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+   a) under this License, provided that you make a good faith effort to
+   ensure that, in the event an Application does not supply the
+   function or data, the facility still operates, and performs
+   whatever part of its purpose remains meaningful, or
+
+   b) under the GNU GPL, with none of the additional permissions of
+   this License applicable to that copy.
+
+  3. Object Code Incorporating Material from Library Header Files.
+
+  The object code form of an Application may incorporate material from
+a header file that is part of the Library.  You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+   a) Give prominent notice with each copy of the object code that the
+   Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the object code with a copy of the GNU GPL and this license
+   document.
+
+  4. Combined Works.
+
+  You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+   a) Give prominent notice with each copy of the Combined Work that
+   the Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the Combined Work with a copy of the GNU GPL and this license
+   document.
+
+   c) For a Combined Work that displays copyright notices during
+   execution, include the copyright notice for the Library among
+   these notices, as well as a reference directing the user to the
+   copies of the GNU GPL and this license document.
+
+   d) Do one of the following:
+
+       0) Convey the Minimal Corresponding Source under the terms of this
+       License, and the Corresponding Application Code in a form
+       suitable for, and under terms that permit, the user to
+       recombine or relink the Application with a modified version of
+       the Linked Version to produce a modified Combined Work, in the
+       manner specified by section 6 of the GNU GPL for conveying
+       Corresponding Source.
+
+       1) Use a suitable shared library mechanism for linking with the
+       Library.  A suitable mechanism is one that (a) uses at run time
+       a copy of the Library already present on the user's computer
+       system, and (b) will operate properly with a modified version
+       of the Library that is interface-compatible with the Linked
+       Version.
+
+   e) Provide Installation Information, but only if you would otherwise
+   be required to provide such information under section 6 of the
+   GNU GPL, and only to the extent that such information is
+   necessary to install and execute a modified version of the
+   Combined Work produced by recombining or relinking the
+   Application with a modified version of the Linked Version. (If
+   you use option 4d0, the Installation Information must accompany
+   the Minimal Corresponding Source and Corresponding Application
+   Code. If you use option 4d1, you must provide the Installation
+   Information in the manner specified by section 6 of the GNU GPL
+   for conveying Corresponding Source.)
+
+  5. Combined Libraries.
+
+  You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+   a) Accompany the combined library with a copy of the same work based
+   on the Library, uncombined with any other library facilities,
+   conveyed under the terms of this License.
+
+   b) Give prominent notice with the combined library that part of it
+   is a work based on the Library, and explaining where to find the
+   accompanying uncombined form of the same work.
+
+  6. Revised Versions of the GNU Lesser General Public License.
+
+  The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+  Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+  If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.

data/Manifest

@@ -0,0 +1,10 @@
+CHANGELOG
+lib/bounded_async_task_runner.rb
+lib/parallel_each.rb
+LICENSE
+Manifest
+Rakefile
+README.markdown
+spec/bounded_async_task_runner_spec.rb
+spec/parallel_each_spec.rb
+spec/spec_helper.rb

data/README.markdown

@@ -0,0 +1,70 @@
+Parallel Each
+========
+
+Parallel each allows you to iterated through any `Enumerable` handling
+the items in parallel.  
+
+    results = []
+    [1,2,3,4].p_each do |i|
+      results << i
+    end
+    results   # => [3,2,4,1]
+
+The parallel processing happens in threads The given block is executed
+for each item in the array in a separate thread.  Obviously, this
+could get dodgy for large `Enumerable`s.  To handle that `#p_each`
+will never have more than a set number of threads running
+simultaneously.  The default limit is 20 but you can change that by
+passing a number to `#p_each`.
+
+    results = []
+    [1,2,3,4].p_each(2) do |i|
+      sleep 1
+      results << i
+    end
+    results   # => [2,1,3,4]
+    
+
+Parallel iterates through the items in order getting ahead of what can
+be processed immediately.  This mean that you can use `#p_each` on
+very large lazy loaded lists (such as the results of
+`ActiveRecord#find`) without instantiating every item in the list at
+once.
+
+Performance
+-------
+
+The performance characteristics of parallel iteration varies greatly
+depending on the platform and work being performed.
+
+I/O bound work is where `#p_each` will really shine.  In this
+situation the overall processing time will, generally, be improved
+regardless of the version/flavor of Ruby that is being used or the
+number of CPUs.
+
+If the work is CPU bound the impacts of parallelization will depend on
+version/flavor of Ruby and the number of CPUs.  With more than one CPU
+running Ruby 1.9 or JRuby should result in an improvement of the
+overall processing time.  With one CPU or when running Ruby 1.8 there
+will be no improvement (and possibly a slight degradation) of the
+overall processing time.
+
+Legalese
+-----
+
+Copyright 2009 Peter Williams.
+
+This Program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or (at
+your option) any later version.
+
+This program is distributed in the hope that it will be useful, but
+WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+General Public License for more details.
+
+You should have received a copy of the GNU Lesser General Public
+License along with this program.  If not, see
+<http://www.gnu.org/licenses/>.
+

data/Rakefile

@@ -0,0 +1,24 @@
+require 'rubygems'
+require 'rake'
+require 'spec/rake/spectask'
+require 'echoe'
+require 'yard'
+
+Spec::Rake::SpecTask.new
+
+Echoe.new('parallel-each') do |p|
+  p.description     = "Parallel iteration for Enumerable"
+  p.url             = "http://github.com/pezra/parallel-each"
+  p.author          = "Peter Williams"
+  p.email           = "pezra@barelyenough.org"
+  p.retain_gemspec  = true
+end
+
+
+desc "Generate Yardoc"
+YARD::Rake::YardocTask.new
+
+desc "Update rubyforge documentation"
+task :update_docs => :yardoc do
+  puts %x{rsync -aPz doc/* pezra@parallel-each.rubyforge.org:/var/www/gforge-projects/parallel-each/}
+end

data/lib/bounded_async_task_runner.rb

@@ -0,0 +1,86 @@
+class AsyncTaskError < Exception
+  attr_reader :cause
+  
+  def initialize(exception)
+    @cause = exception
+    super("Worker error: #{cause.message} (#{cause.class.name})")
+  end
+end
+
+
+class BoundedAsyncTaskRunner
+  attr_reader :max_workers
+
+  # Create a new BoundedAsyncTaskRunner.
+  #
+  # @param [Integer] max_workers the maximum number of asynchronous
+  #   tasks to simultaneously execute.
+  def initialize(max_workers=20)
+    @threads = Array.new
+    @max_workers = max_workers
+  end
+
+  # Execute the provided block asynchronously.  
+  #
+  # @param *args Objects to yield to the provided block when it is
+  #   invoked.
+  #
+  # @yield The task to execute asynchronously.
+  #
+  # @yieldparam *args The arguments passed into this method.
+  #
+  # @raise [AsyncTaskError] If any previously executed task has failed
+  #   (i.e. raised an error) an AsyncTaskError will be raised instead
+  #   of executing the async task.
+  def do(*args, &block)
+    purge_dead
+    until @threads.size < max_workers
+      purge_dead
+      break if @threads.size < max_workers
+      # We have still have the maximum number of active threads so we
+      # need to wait for one to finish.
+
+      @threads.first.join(0.1)  # Join on the oldest thread because it will probably finish first.
+    end
+    # We can start new thread without exceeding the threshold
+
+    thread_args = [block] + args
+
+    @threads << Thread.start([block] + args) do |args|
+      task = args.shift
+      
+      begin
+        task.call(*args)
+      rescue Exception => e
+        AsyncTaskError.new(e)
+      end
+    end
+  end
+  alias :do_async :do
+
+
+  # Blocks until all currently running tasks to complete.
+  #
+  # @raise [AsyncTaskError] If any previously executed task has failed
+  #   (i.e. raised an error) an AsyncTaskError will be raised.
+  def wait_for_all_to_finish
+    until @threads.empty?
+      @threads.first.join
+      purge_dead
+    end    
+  end
+
+  private
+  def purge_dead
+    @threads.each do |t|
+      next if t.alive?  # ignore the alive ones
+      
+      if t.value.kind_of?(AsyncTaskError)
+        raise(t.value)
+      else
+        @threads.delete(t)
+      end
+    end
+  end
+
+end

data/lib/parallel_each.rb

@@ -0,0 +1,14 @@
+require 'pathname'
+require Pathname(__FILE__).dirname + './bounded_async_task_runner'
+
+module Enumerable
+  def p_each(max_tasks=20, &block)
+    runner = BoundedAsyncTaskRunner.new(max_tasks)
+
+    each do |item|
+      runner.do(item, &block)
+    end
+
+    runner.wait_for_all_to_finish
+  end
+end

data/parallel-each.gemspec

@@ -0,0 +1,31 @@
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+  s.name = %q{parallel-each}
+  s.version = "1.0.0"
+
+  s.required_rubygems_version = Gem::Requirement.new(">= 1.2") if s.respond_to? :required_rubygems_version=
+  s.authors = ["Peter Williams"]
+  s.date = %q{2009-08-27}
+  s.description = %q{Parallel iteration for Enumerable}
+  s.email = %q{pezra@barelyenough.org}
+  s.extra_rdoc_files = ["CHANGELOG", "lib/bounded_async_task_runner.rb", "lib/parallel_each.rb", "LICENSE", "README.markdown"]
+  s.files = ["CHANGELOG", "lib/bounded_async_task_runner.rb", "lib/parallel_each.rb", "LICENSE", "Manifest", "Rakefile", "README.markdown", "spec/bounded_async_task_runner_spec.rb", "spec/parallel_each_spec.rb", "spec/spec_helper.rb", "parallel-each.gemspec"]
+  s.has_rdoc = true
+  s.homepage = %q{http://github.com/pezra/parallel-each}
+  s.rdoc_options = ["--line-numbers", "--inline-source", "--title", "Parallel-each", "--main", "README.markdown"]
+  s.require_paths = ["lib"]
+  s.rubyforge_project = %q{parallel-each}
+  s.rubygems_version = %q{1.3.1}
+  s.summary = %q{Parallel iteration for Enumerable}
+
+  if s.respond_to? :specification_version then
+    current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
+    s.specification_version = 2
+
+    if Gem::Version.new(Gem::RubyGemsVersion) >= Gem::Version.new('1.2.0') then
+    else
+    end
+  else
+  end
+end

data/spec/bounded_async_task_runner_spec.rb

@@ -0,0 +1,65 @@
+require 'pathname'
+require Pathname(__FILE__).dirname + 'spec_helper'
+
+require 'bounded_async_task_runner'
+
+describe BoundedAsyncTaskRunner do
+  before do
+    @runner = BoundedAsyncTaskRunner.new(5)
+  end
+
+  describe "#do(&block)" do
+    it "should run tasks" do
+      out = Array.new
+
+      @runner.do(out) {|out| out << :marker}
+      @runner.wait_for_all_to_finish
+
+      out.should == [:marker]
+    end 
+
+    it "should run jobs as the same time" do
+      out = Array.new
+      
+      @runner.do(out) {|out| sleep(0.1); out << :a}
+      @runner.do(out) {|out| out << :b}
+      
+      @runner.wait_for_all_to_finish
+      out.should == [:b, :a]
+    end 
+
+    it "should raise exception if any tasks fail" do
+      lambda {
+        @runner.do { raise ArgumentError, "testing" }
+        @runner.wait_for_all_to_finish
+      }.should raise_error(AsyncTaskError, "Worker error: testing (ArgumentError)")
+    end 
+
+    it "should included exception raised by task AsyncTaskError exception" do
+      exception = ArgumentError.new("testing")
+      @runner.do(exception) { raise(exception) }
+      begin
+        @runner.wait_for_all_to_finish
+      rescue AsyncTaskError => e
+        e.cause.should == exception
+      end
+    end 
+
+    it "should not run more than the max number of tasks at any one moment" do
+      # I don't know how to test this...
+    end 
+  end
+
+  describe "#wait_for_all_to_finish" do
+    it "should not return until all tasks are finished" do
+      out = Array.new
+      
+      3.times do 
+        @runner.do(out) {|out| sleep(0.1); out << :a}
+      end
+      @runner.wait_for_all_to_finish
+
+      out.should == [:a, :a, :a]
+    end 
+  end 
+end 

data/spec/parallel_each_spec.rb

@@ -0,0 +1,16 @@
+require File.dirname(__FILE__) + "/spec_helper"
+require 'parallel_each'
+
+describe '#p_each' do 
+  it "should execute block for each item" do 
+    i = 0
+    %w(this that).p_each {i+=1}
+    i.should eql(2)
+  end
+
+  it "should execute items in parallel" do
+    results = []
+    [2, 1].p_each{|i| sleep i; results << i}
+    results.should eql([1, 2])
+  end
+end

data/spec/spec_helper.rb

@@ -0,0 +1,9 @@
+require 'rubygems'
+require 'pathname'
+require 'spec'
+
+$LOAD_PATH.unshift Pathname(__FILE__).dirname + '../lib'
+
+Spec::Runner.configure do |config|
+
+end

metadata

@@ -0,0 +1,72 @@
+--- !ruby/object:Gem::Specification 
+name: parallel-each
+version: !ruby/object:Gem::Version 
+  version: 1.0.0
+platform: ruby
+authors: 
+- Peter Williams
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2009-08-27 00:00:00 -06:00
+default_executable: 
+dependencies: []
+
+description: Parallel iteration for Enumerable
+email: pezra@barelyenough.org
+executables: []
+
+extensions: []
+
+extra_rdoc_files: 
+- CHANGELOG
+- lib/bounded_async_task_runner.rb
+- lib/parallel_each.rb
+- LICENSE
+- README.markdown
+files: 
+- CHANGELOG
+- lib/bounded_async_task_runner.rb
+- lib/parallel_each.rb
+- LICENSE
+- Manifest
+- Rakefile
+- README.markdown
+- spec/bounded_async_task_runner_spec.rb
+- spec/parallel_each_spec.rb
+- spec/spec_helper.rb
+- parallel-each.gemspec
+has_rdoc: true
+homepage: http://github.com/pezra/parallel-each
+post_install_message: 
+rdoc_options: 
+- --line-numbers
+- --inline-source
+- --title
+- Parallel-each
+- --main
+- README.markdown
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "1.2"
+  version: 
+requirements: []
+
+rubyforge_project: parallel-each
+rubygems_version: 1.3.1
+signing_key: 
+specification_version: 2
+summary: Parallel iteration for Enumerable
+test_files: []
+