parallel-each 1.0.0

Sign up to get free protection for your applications and to get access to all the features.
@@ -0,0 +1,5 @@
1
+ v1.0.0
2
+ =====
3
+
4
+ * Parallel iteration over item in an Enumerable
5
+
data/LICENSE ADDED
@@ -0,0 +1,165 @@
1
+ GNU LESSER GENERAL PUBLIC LICENSE
2
+ Version 3, 29 June 2007
3
+
4
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
5
+ Everyone is permitted to copy and distribute verbatim copies
6
+ of this license document, but changing it is not allowed.
7
+
8
+
9
+ This version of the GNU Lesser General Public License incorporates
10
+ the terms and conditions of version 3 of the GNU General Public
11
+ License, supplemented by the additional permissions listed below.
12
+
13
+ 0. Additional Definitions.
14
+
15
+ As used herein, "this License" refers to version 3 of the GNU Lesser
16
+ General Public License, and the "GNU GPL" refers to version 3 of the GNU
17
+ General Public License.
18
+
19
+ "The Library" refers to a covered work governed by this License,
20
+ other than an Application or a Combined Work as defined below.
21
+
22
+ An "Application" is any work that makes use of an interface provided
23
+ by the Library, but which is not otherwise based on the Library.
24
+ Defining a subclass of a class defined by the Library is deemed a mode
25
+ of using an interface provided by the Library.
26
+
27
+ A "Combined Work" is a work produced by combining or linking an
28
+ Application with the Library. The particular version of the Library
29
+ with which the Combined Work was made is also called the "Linked
30
+ Version".
31
+
32
+ The "Minimal Corresponding Source" for a Combined Work means the
33
+ Corresponding Source for the Combined Work, excluding any source code
34
+ for portions of the Combined Work that, considered in isolation, are
35
+ based on the Application, and not on the Linked Version.
36
+
37
+ The "Corresponding Application Code" for a Combined Work means the
38
+ object code and/or source code for the Application, including any data
39
+ and utility programs needed for reproducing the Combined Work from the
40
+ Application, but excluding the System Libraries of the Combined Work.
41
+
42
+ 1. Exception to Section 3 of the GNU GPL.
43
+
44
+ You may convey a covered work under sections 3 and 4 of this License
45
+ without being bound by section 3 of the GNU GPL.
46
+
47
+ 2. Conveying Modified Versions.
48
+
49
+ If you modify a copy of the Library, and, in your modifications, a
50
+ facility refers to a function or data to be supplied by an Application
51
+ that uses the facility (other than as an argument passed when the
52
+ facility is invoked), then you may convey a copy of the modified
53
+ version:
54
+
55
+ a) under this License, provided that you make a good faith effort to
56
+ ensure that, in the event an Application does not supply the
57
+ function or data, the facility still operates, and performs
58
+ whatever part of its purpose remains meaningful, or
59
+
60
+ b) under the GNU GPL, with none of the additional permissions of
61
+ this License applicable to that copy.
62
+
63
+ 3. Object Code Incorporating Material from Library Header Files.
64
+
65
+ The object code form of an Application may incorporate material from
66
+ a header file that is part of the Library. You may convey such object
67
+ code under terms of your choice, provided that, if the incorporated
68
+ material is not limited to numerical parameters, data structure
69
+ layouts and accessors, or small macros, inline functions and templates
70
+ (ten or fewer lines in length), you do both of the following:
71
+
72
+ a) Give prominent notice with each copy of the object code that the
73
+ Library is used in it and that the Library and its use are
74
+ covered by this License.
75
+
76
+ b) Accompany the object code with a copy of the GNU GPL and this license
77
+ document.
78
+
79
+ 4. Combined Works.
80
+
81
+ You may convey a Combined Work under terms of your choice that,
82
+ taken together, effectively do not restrict modification of the
83
+ portions of the Library contained in the Combined Work and reverse
84
+ engineering for debugging such modifications, if you also do each of
85
+ the following:
86
+
87
+ a) Give prominent notice with each copy of the Combined Work that
88
+ the Library is used in it and that the Library and its use are
89
+ covered by this License.
90
+
91
+ b) Accompany the Combined Work with a copy of the GNU GPL and this license
92
+ document.
93
+
94
+ c) For a Combined Work that displays copyright notices during
95
+ execution, include the copyright notice for the Library among
96
+ these notices, as well as a reference directing the user to the
97
+ copies of the GNU GPL and this license document.
98
+
99
+ d) Do one of the following:
100
+
101
+ 0) Convey the Minimal Corresponding Source under the terms of this
102
+ License, and the Corresponding Application Code in a form
103
+ suitable for, and under terms that permit, the user to
104
+ recombine or relink the Application with a modified version of
105
+ the Linked Version to produce a modified Combined Work, in the
106
+ manner specified by section 6 of the GNU GPL for conveying
107
+ Corresponding Source.
108
+
109
+ 1) Use a suitable shared library mechanism for linking with the
110
+ Library. A suitable mechanism is one that (a) uses at run time
111
+ a copy of the Library already present on the user's computer
112
+ system, and (b) will operate properly with a modified version
113
+ of the Library that is interface-compatible with the Linked
114
+ Version.
115
+
116
+ e) Provide Installation Information, but only if you would otherwise
117
+ be required to provide such information under section 6 of the
118
+ GNU GPL, and only to the extent that such information is
119
+ necessary to install and execute a modified version of the
120
+ Combined Work produced by recombining or relinking the
121
+ Application with a modified version of the Linked Version. (If
122
+ you use option 4d0, the Installation Information must accompany
123
+ the Minimal Corresponding Source and Corresponding Application
124
+ Code. If you use option 4d1, you must provide the Installation
125
+ Information in the manner specified by section 6 of the GNU GPL
126
+ for conveying Corresponding Source.)
127
+
128
+ 5. Combined Libraries.
129
+
130
+ You may place library facilities that are a work based on the
131
+ Library side by side in a single library together with other library
132
+ facilities that are not Applications and are not covered by this
133
+ License, and convey such a combined library under terms of your
134
+ choice, if you do both of the following:
135
+
136
+ a) Accompany the combined library with a copy of the same work based
137
+ on the Library, uncombined with any other library facilities,
138
+ conveyed under the terms of this License.
139
+
140
+ b) Give prominent notice with the combined library that part of it
141
+ is a work based on the Library, and explaining where to find the
142
+ accompanying uncombined form of the same work.
143
+
144
+ 6. Revised Versions of the GNU Lesser General Public License.
145
+
146
+ The Free Software Foundation may publish revised and/or new versions
147
+ of the GNU Lesser General Public License from time to time. Such new
148
+ versions will be similar in spirit to the present version, but may
149
+ differ in detail to address new problems or concerns.
150
+
151
+ Each version is given a distinguishing version number. If the
152
+ Library as you received it specifies that a certain numbered version
153
+ of the GNU Lesser General Public License "or any later version"
154
+ applies to it, you have the option of following the terms and
155
+ conditions either of that published version or of any later version
156
+ published by the Free Software Foundation. If the Library as you
157
+ received it does not specify a version number of the GNU Lesser
158
+ General Public License, you may choose any version of the GNU Lesser
159
+ General Public License ever published by the Free Software Foundation.
160
+
161
+ If the Library as you received it specifies that a proxy can decide
162
+ whether future versions of the GNU Lesser General Public License shall
163
+ apply, that proxy's public statement of acceptance of any version is
164
+ permanent authorization for you to choose that version for the
165
+ Library.
@@ -0,0 +1,10 @@
1
+ CHANGELOG
2
+ lib/bounded_async_task_runner.rb
3
+ lib/parallel_each.rb
4
+ LICENSE
5
+ Manifest
6
+ Rakefile
7
+ README.markdown
8
+ spec/bounded_async_task_runner_spec.rb
9
+ spec/parallel_each_spec.rb
10
+ spec/spec_helper.rb
@@ -0,0 +1,70 @@
1
+ Parallel Each
2
+ ========
3
+
4
+ Parallel each allows you to iterated through any `Enumerable` handling
5
+ the items in parallel.
6
+
7
+ results = []
8
+ [1,2,3,4].p_each do |i|
9
+ results << i
10
+ end
11
+ results # => [3,2,4,1]
12
+
13
+ The parallel processing happens in threads The given block is executed
14
+ for each item in the array in a separate thread. Obviously, this
15
+ could get dodgy for large `Enumerable`s. To handle that `#p_each`
16
+ will never have more than a set number of threads running
17
+ simultaneously. The default limit is 20 but you can change that by
18
+ passing a number to `#p_each`.
19
+
20
+ results = []
21
+ [1,2,3,4].p_each(2) do |i|
22
+ sleep 1
23
+ results << i
24
+ end
25
+ results # => [2,1,3,4]
26
+
27
+
28
+ Parallel iterates through the items in order getting ahead of what can
29
+ be processed immediately. This mean that you can use `#p_each` on
30
+ very large lazy loaded lists (such as the results of
31
+ `ActiveRecord#find`) without instantiating every item in the list at
32
+ once.
33
+
34
+ Performance
35
+ -------
36
+
37
+ The performance characteristics of parallel iteration varies greatly
38
+ depending on the platform and work being performed.
39
+
40
+ I/O bound work is where `#p_each` will really shine. In this
41
+ situation the overall processing time will, generally, be improved
42
+ regardless of the version/flavor of Ruby that is being used or the
43
+ number of CPUs.
44
+
45
+ If the work is CPU bound the impacts of parallelization will depend on
46
+ version/flavor of Ruby and the number of CPUs. With more than one CPU
47
+ running Ruby 1.9 or JRuby should result in an improvement of the
48
+ overall processing time. With one CPU or when running Ruby 1.8 there
49
+ will be no improvement (and possibly a slight degradation) of the
50
+ overall processing time.
51
+
52
+ Legalese
53
+ -----
54
+
55
+ Copyright 2009 Peter Williams.
56
+
57
+ This Program is free software: you can redistribute it and/or modify
58
+ it under the terms of the GNU General Public License as published by
59
+ the Free Software Foundation, either version 3 of the License, or (at
60
+ your option) any later version.
61
+
62
+ This program is distributed in the hope that it will be useful, but
63
+ WITHOUT ANY WARRANTY; without even the implied warranty of
64
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
65
+ General Public License for more details.
66
+
67
+ You should have received a copy of the GNU Lesser General Public
68
+ License along with this program. If not, see
69
+ <http://www.gnu.org/licenses/>.
70
+
@@ -0,0 +1,24 @@
1
+ require 'rubygems'
2
+ require 'rake'
3
+ require 'spec/rake/spectask'
4
+ require 'echoe'
5
+ require 'yard'
6
+
7
+ Spec::Rake::SpecTask.new
8
+
9
+ Echoe.new('parallel-each') do |p|
10
+ p.description = "Parallel iteration for Enumerable"
11
+ p.url = "http://github.com/pezra/parallel-each"
12
+ p.author = "Peter Williams"
13
+ p.email = "pezra@barelyenough.org"
14
+ p.retain_gemspec = true
15
+ end
16
+
17
+
18
+ desc "Generate Yardoc"
19
+ YARD::Rake::YardocTask.new
20
+
21
+ desc "Update rubyforge documentation"
22
+ task :update_docs => :yardoc do
23
+ puts %x{rsync -aPz doc/* pezra@parallel-each.rubyforge.org:/var/www/gforge-projects/parallel-each/}
24
+ end
@@ -0,0 +1,86 @@
1
+ class AsyncTaskError < Exception
2
+ attr_reader :cause
3
+
4
+ def initialize(exception)
5
+ @cause = exception
6
+ super("Worker error: #{cause.message} (#{cause.class.name})")
7
+ end
8
+ end
9
+
10
+
11
+ class BoundedAsyncTaskRunner
12
+ attr_reader :max_workers
13
+
14
+ # Create a new BoundedAsyncTaskRunner.
15
+ #
16
+ # @param [Integer] max_workers the maximum number of asynchronous
17
+ # tasks to simultaneously execute.
18
+ def initialize(max_workers=20)
19
+ @threads = Array.new
20
+ @max_workers = max_workers
21
+ end
22
+
23
+ # Execute the provided block asynchronously.
24
+ #
25
+ # @param *args Objects to yield to the provided block when it is
26
+ # invoked.
27
+ #
28
+ # @yield The task to execute asynchronously.
29
+ #
30
+ # @yieldparam *args The arguments passed into this method.
31
+ #
32
+ # @raise [AsyncTaskError] If any previously executed task has failed
33
+ # (i.e. raised an error) an AsyncTaskError will be raised instead
34
+ # of executing the async task.
35
+ def do(*args, &block)
36
+ purge_dead
37
+ until @threads.size < max_workers
38
+ purge_dead
39
+ break if @threads.size < max_workers
40
+ # We have still have the maximum number of active threads so we
41
+ # need to wait for one to finish.
42
+
43
+ @threads.first.join(0.1) # Join on the oldest thread because it will probably finish first.
44
+ end
45
+ # We can start new thread without exceeding the threshold
46
+
47
+ thread_args = [block] + args
48
+
49
+ @threads << Thread.start([block] + args) do |args|
50
+ task = args.shift
51
+
52
+ begin
53
+ task.call(*args)
54
+ rescue Exception => e
55
+ AsyncTaskError.new(e)
56
+ end
57
+ end
58
+ end
59
+ alias :do_async :do
60
+
61
+
62
+ # Blocks until all currently running tasks to complete.
63
+ #
64
+ # @raise [AsyncTaskError] If any previously executed task has failed
65
+ # (i.e. raised an error) an AsyncTaskError will be raised.
66
+ def wait_for_all_to_finish
67
+ until @threads.empty?
68
+ @threads.first.join
69
+ purge_dead
70
+ end
71
+ end
72
+
73
+ private
74
+ def purge_dead
75
+ @threads.each do |t|
76
+ next if t.alive? # ignore the alive ones
77
+
78
+ if t.value.kind_of?(AsyncTaskError)
79
+ raise(t.value)
80
+ else
81
+ @threads.delete(t)
82
+ end
83
+ end
84
+ end
85
+
86
+ end
@@ -0,0 +1,14 @@
1
+ require 'pathname'
2
+ require Pathname(__FILE__).dirname + './bounded_async_task_runner'
3
+
4
+ module Enumerable
5
+ def p_each(max_tasks=20, &block)
6
+ runner = BoundedAsyncTaskRunner.new(max_tasks)
7
+
8
+ each do |item|
9
+ runner.do(item, &block)
10
+ end
11
+
12
+ runner.wait_for_all_to_finish
13
+ end
14
+ end
@@ -0,0 +1,31 @@
1
+ # -*- encoding: utf-8 -*-
2
+
3
+ Gem::Specification.new do |s|
4
+ s.name = %q{parallel-each}
5
+ s.version = "1.0.0"
6
+
7
+ s.required_rubygems_version = Gem::Requirement.new(">= 1.2") if s.respond_to? :required_rubygems_version=
8
+ s.authors = ["Peter Williams"]
9
+ s.date = %q{2009-08-27}
10
+ s.description = %q{Parallel iteration for Enumerable}
11
+ s.email = %q{pezra@barelyenough.org}
12
+ s.extra_rdoc_files = ["CHANGELOG", "lib/bounded_async_task_runner.rb", "lib/parallel_each.rb", "LICENSE", "README.markdown"]
13
+ s.files = ["CHANGELOG", "lib/bounded_async_task_runner.rb", "lib/parallel_each.rb", "LICENSE", "Manifest", "Rakefile", "README.markdown", "spec/bounded_async_task_runner_spec.rb", "spec/parallel_each_spec.rb", "spec/spec_helper.rb", "parallel-each.gemspec"]
14
+ s.has_rdoc = true
15
+ s.homepage = %q{http://github.com/pezra/parallel-each}
16
+ s.rdoc_options = ["--line-numbers", "--inline-source", "--title", "Parallel-each", "--main", "README.markdown"]
17
+ s.require_paths = ["lib"]
18
+ s.rubyforge_project = %q{parallel-each}
19
+ s.rubygems_version = %q{1.3.1}
20
+ s.summary = %q{Parallel iteration for Enumerable}
21
+
22
+ if s.respond_to? :specification_version then
23
+ current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
24
+ s.specification_version = 2
25
+
26
+ if Gem::Version.new(Gem::RubyGemsVersion) >= Gem::Version.new('1.2.0') then
27
+ else
28
+ end
29
+ else
30
+ end
31
+ end
@@ -0,0 +1,65 @@
1
+ require 'pathname'
2
+ require Pathname(__FILE__).dirname + 'spec_helper'
3
+
4
+ require 'bounded_async_task_runner'
5
+
6
+ describe BoundedAsyncTaskRunner do
7
+ before do
8
+ @runner = BoundedAsyncTaskRunner.new(5)
9
+ end
10
+
11
+ describe "#do(&block)" do
12
+ it "should run tasks" do
13
+ out = Array.new
14
+
15
+ @runner.do(out) {|out| out << :marker}
16
+ @runner.wait_for_all_to_finish
17
+
18
+ out.should == [:marker]
19
+ end
20
+
21
+ it "should run jobs as the same time" do
22
+ out = Array.new
23
+
24
+ @runner.do(out) {|out| sleep(0.1); out << :a}
25
+ @runner.do(out) {|out| out << :b}
26
+
27
+ @runner.wait_for_all_to_finish
28
+ out.should == [:b, :a]
29
+ end
30
+
31
+ it "should raise exception if any tasks fail" do
32
+ lambda {
33
+ @runner.do { raise ArgumentError, "testing" }
34
+ @runner.wait_for_all_to_finish
35
+ }.should raise_error(AsyncTaskError, "Worker error: testing (ArgumentError)")
36
+ end
37
+
38
+ it "should included exception raised by task AsyncTaskError exception" do
39
+ exception = ArgumentError.new("testing")
40
+ @runner.do(exception) { raise(exception) }
41
+ begin
42
+ @runner.wait_for_all_to_finish
43
+ rescue AsyncTaskError => e
44
+ e.cause.should == exception
45
+ end
46
+ end
47
+
48
+ it "should not run more than the max number of tasks at any one moment" do
49
+ # I don't know how to test this...
50
+ end
51
+ end
52
+
53
+ describe "#wait_for_all_to_finish" do
54
+ it "should not return until all tasks are finished" do
55
+ out = Array.new
56
+
57
+ 3.times do
58
+ @runner.do(out) {|out| sleep(0.1); out << :a}
59
+ end
60
+ @runner.wait_for_all_to_finish
61
+
62
+ out.should == [:a, :a, :a]
63
+ end
64
+ end
65
+ end
@@ -0,0 +1,16 @@
1
+ require File.dirname(__FILE__) + "/spec_helper"
2
+ require 'parallel_each'
3
+
4
+ describe '#p_each' do
5
+ it "should execute block for each item" do
6
+ i = 0
7
+ %w(this that).p_each {i+=1}
8
+ i.should eql(2)
9
+ end
10
+
11
+ it "should execute items in parallel" do
12
+ results = []
13
+ [2, 1].p_each{|i| sleep i; results << i}
14
+ results.should eql([1, 2])
15
+ end
16
+ end
@@ -0,0 +1,9 @@
1
+ require 'rubygems'
2
+ require 'pathname'
3
+ require 'spec'
4
+
5
+ $LOAD_PATH.unshift Pathname(__FILE__).dirname + '../lib'
6
+
7
+ Spec::Runner.configure do |config|
8
+
9
+ end
metadata ADDED
@@ -0,0 +1,72 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: parallel-each
3
+ version: !ruby/object:Gem::Version
4
+ version: 1.0.0
5
+ platform: ruby
6
+ authors:
7
+ - Peter Williams
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+
12
+ date: 2009-08-27 00:00:00 -06:00
13
+ default_executable:
14
+ dependencies: []
15
+
16
+ description: Parallel iteration for Enumerable
17
+ email: pezra@barelyenough.org
18
+ executables: []
19
+
20
+ extensions: []
21
+
22
+ extra_rdoc_files:
23
+ - CHANGELOG
24
+ - lib/bounded_async_task_runner.rb
25
+ - lib/parallel_each.rb
26
+ - LICENSE
27
+ - README.markdown
28
+ files:
29
+ - CHANGELOG
30
+ - lib/bounded_async_task_runner.rb
31
+ - lib/parallel_each.rb
32
+ - LICENSE
33
+ - Manifest
34
+ - Rakefile
35
+ - README.markdown
36
+ - spec/bounded_async_task_runner_spec.rb
37
+ - spec/parallel_each_spec.rb
38
+ - spec/spec_helper.rb
39
+ - parallel-each.gemspec
40
+ has_rdoc: true
41
+ homepage: http://github.com/pezra/parallel-each
42
+ post_install_message:
43
+ rdoc_options:
44
+ - --line-numbers
45
+ - --inline-source
46
+ - --title
47
+ - Parallel-each
48
+ - --main
49
+ - README.markdown
50
+ require_paths:
51
+ - lib
52
+ required_ruby_version: !ruby/object:Gem::Requirement
53
+ requirements:
54
+ - - ">="
55
+ - !ruby/object:Gem::Version
56
+ version: "0"
57
+ version:
58
+ required_rubygems_version: !ruby/object:Gem::Requirement
59
+ requirements:
60
+ - - ">="
61
+ - !ruby/object:Gem::Version
62
+ version: "1.2"
63
+ version:
64
+ requirements: []
65
+
66
+ rubyforge_project: parallel-each
67
+ rubygems_version: 1.3.1
68
+ signing_key:
69
+ specification_version: 2
70
+ summary: Parallel iteration for Enumerable
71
+ test_files: []
72
+