async_enum 0.0.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,15 @@
1
+ ---
2
+ !binary "U0hBMQ==":
3
+ metadata.gz: !binary |-
4
+ MWU0MWNjY2UwYTE3OTI5Njc1OThiZTAyMDNjMjliNmQ3MWMzZmZjMg==
5
+ data.tar.gz: !binary |-
6
+ NGM5ZDA0NjA5NTI3M2YyNjVjZTA1N2UxOTVmYzI5MGQ2OGQ5YzhlNg==
7
+ !binary "U0hBNTEy":
8
+ metadata.gz: !binary |-
9
+ MjQ0Nzc5NmVlNTQzMzMzZmEyNzBmOTU4ZThkYWZmOWYwYmFmYTUxMDkxMTU5
10
+ NTFhNzI0MTUxNDQ2NGIyZjU3OTAyMTFiZTFiZjdmZjNlYWNhMGVlYjdkNTY2
11
+ OGFmZWM4OTk4MzUyNGUwYTQ3YWQ1N2Q3YjZjYWU2NjFkMmNlNTc=
12
+ data.tar.gz: !binary |-
13
+ MWQyYWM1ZWQ1ODc1YjQ0M2M2Y2VhMjQzM2I5ODhiODU0MjEzOTdmNzM0ZTY2
14
+ ZDcwMTE5YjhlYTEwZmUwMDBlMmE5MTI0MWY4MGU3ZWNlOTMxMzlhN2ZhOWIy
15
+ M2ExYjE1ZWI2MjM0MjBkMGM3ZTY1YWFiNDhjNmIwNmNmZWYzOTk=
data/.gitignore ADDED
@@ -0,0 +1,35 @@
1
+ #********** osx template**********
2
+
3
+ .DS_Store
4
+
5
+ # Thumbnails
6
+ ._*
7
+
8
+ # Files that might appear on external disk
9
+ .Spotlight-V100
10
+ .Trashes
11
+
12
+
13
+ #********** ruby template**********
14
+
15
+ *.gem
16
+ Gemfile.lock
17
+ *.rbc
18
+ .bundle
19
+ .config
20
+ coverage
21
+ InstalledFiles
22
+ lib/bundler/man
23
+ pkg
24
+ rdoc
25
+ spec/reports
26
+ test/tmp
27
+ test/version_tmp
28
+ tmp
29
+ bin/
30
+
31
+ # YARD artifacts
32
+ .yardoc
33
+ _yardoc
34
+ doc/
35
+
data/.travis.yml ADDED
@@ -0,0 +1,4 @@
1
+ language: ruby
2
+ rvm:
3
+ - 2.0.0
4
+ - 1.9.3
data/Gemfile ADDED
@@ -0,0 +1,3 @@
1
+ source 'https://rubygems.org/'
2
+
3
+ gemspec
data/README.md ADDED
@@ -0,0 +1,121 @@
1
+ # async\_enum
2
+
3
+ Inspired by Enumerable#lazy coupled with Enumerator::Lazy, and the Async.js library, I thought it would be cool to have syntax like `urls.async.map{ |url| http_get(url) }`.
4
+
5
+ ![](https://fbcdn-sphotos-b-a.akamaihd.net/hphotos-ak-prn1/554187_10151609082562269_1115589261_n.jpg)
6
+
7
+ #### Runs In Parallel
8
+
9
+ ```ruby
10
+ require 'benchmark'
11
+
12
+ sync = Benchmark.measure do
13
+ 5.times.each{ sleep 0.1 }
14
+ end
15
+
16
+ async = Benchmark.measure do
17
+ 5.times.async.each{ sleep 0.1 }
18
+ end
19
+
20
+ puts sync, async
21
+ # 0.000000 0.000000 0.000000 ( 0.500782)
22
+ # 0.000000 0.010000 0.010000 ( 0.102763)
23
+ ```
24
+
25
+ #### How It Works
26
+
27
+ The implementation was based on `Enumerable#lazy` introduced with Ruby 2.0. `Enumerator::Async` follows a similar approach, where the Enumerator is passed into the constructor.
28
+
29
+ ```ruby
30
+ enum = ('a'..'z').each
31
+
32
+ # the following are equivalent
33
+
34
+ Enumerator::Async.new(enum)
35
+ enum.async
36
+ ```
37
+
38
+ To get the enumerator back from the async enumerator, simply call `sync` or `to_enum` like so:
39
+
40
+ ```ruby
41
+ enum.async.sync == enum
42
+ # => true
43
+ ```
44
+
45
+ Every operation on the async enumerator affects the contained enumerator in the same way. For instance:
46
+
47
+ ```ruby
48
+ enum.with_index.to_a
49
+ # => [ ['a', 0], ['b', 1], ['c', 2] ... ]
50
+
51
+ enum.async.with_index.to_a
52
+ # => [ ['a', 0], ['b', 1], ['c', 2] ... ]
53
+ ```
54
+
55
+ Async methods can be chained just like tipical enumerator methods:
56
+
57
+ ```ruby
58
+ enum.async.each{ sleep(0.1) }.each{ sleep(0.1) }
59
+ ```
60
+
61
+ #### How to use it
62
+
63
+ The method `Enumerable#async` was added so that every collection can be processed in parallel:
64
+
65
+ ```ruby
66
+ [ 0, 1, 2 ].async.each{ |i| puts i }
67
+ (0..2).async.map(&:to_i)
68
+
69
+ # or chain to your heart's content
70
+
71
+ (0..5).reject(&:even?).reverse.async.with_index.map{ |x, index| x + index }
72
+ ```
73
+
74
+ #### Limiting thread pool size
75
+
76
+ To limit the thread pool size, you can pass in an optional parameter to `async`. Suppose for performance reasons you want to use a maximum of 4 threads:
77
+
78
+ ```ruby
79
+ (0..100).async(4).each do
80
+ # use bandwith
81
+ end
82
+ ```
83
+
84
+ #### Preventing race conditions
85
+
86
+ When programming concurrently, nasty bugs can come up because some operations aren't atomic. For instance, incrementing a variable `x += 1` will not necessarily work as expected. To provide easy locking, there's a DSL-style `lock` method you can use in the block passed to the async enum.
87
+
88
+ ```ruby
89
+ count = 0
90
+ ('a'..'z').async.each do
91
+ lock :count do
92
+ count += 1
93
+ end
94
+ end
95
+ count
96
+ # => 26
97
+ ```
98
+
99
+ The name of the lock doesn't matter, but using the variable name helps make the code understandable. You should try to use 1 lock per thread-unsafe variable.
100
+
101
+ ## Notes
102
+
103
+ To install it, add it to your gemfile:
104
+
105
+ ```
106
+ # Gemfile
107
+
108
+ gem 'async_enum', github: 'aj0strow/async_enum'
109
+ ```
110
+
111
+ To run the demos, clone the project and run with the library included in the load path:
112
+
113
+ ```
114
+ $ git clone git@github.com:aj0strow/async_enum
115
+ $ cd async_enum
116
+ $ ruby -I lib demos/sleep.rb
117
+ ```
118
+
119
+ **Disclaimer**: I am not an expert at multithreading. Quite the opposite in fact.
120
+
121
+ Please report errors and feel free to contribute and improve things!
data/Rakefile ADDED
@@ -0,0 +1,9 @@
1
+ require 'rake/testtask'
2
+
3
+ Rake::TestTask.new do |t|
4
+ t.libs << "test"
5
+ t.test_files = FileList['test/**/*.rb']
6
+ t.verbose = true
7
+ end
8
+
9
+ task default: :test
@@ -0,0 +1,21 @@
1
+ # coding: utf-8
2
+ lib = File.expand_path('../lib', __FILE__)
3
+ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
+
5
+ Gem::Specification.new do |s|
6
+ s.name = 'async_enum'
7
+ s.version = '0.0.4'
8
+ s.authors = %w(aj0strow)
9
+ s.email = 'alexander.ostrow@gmail.com'
10
+ s.description = 'iterate over enumerable objects concurrently'
11
+ s.summary = 'Async Enumerable and Enumerator'
12
+ s.homepage = 'http://github.com/aj0strow/async_enum'
13
+ s.license = 'MIT'
14
+
15
+ s.files = `git ls-files`.split($/)
16
+ s.test_files = s.files.grep(/test/)
17
+ s.require_paths = %w(lib)
18
+
19
+ s.add_development_dependency 'bundler'
20
+ s.add_development_dependency 'rake'
21
+ end
data/demos/lock.rb ADDED
@@ -0,0 +1,9 @@
1
+ require 'async_enum'
2
+
3
+ count = 0
4
+ ('a'..'z').async.each do
5
+ lock :count do
6
+ count += 1
7
+ end
8
+ end
9
+ puts count
@@ -0,0 +1,12 @@
1
+ require 'benchmark'
2
+ require 'async_enum'
3
+
4
+ default = Benchmark.measure do
5
+ 500.times.async.each{ true }
6
+ end
7
+
8
+ limited = Benchmark.measure do
9
+ 500.times.async(10).each{ true }
10
+ end
11
+
12
+ puts default, limited
data/demos/sleep.rb ADDED
@@ -0,0 +1,12 @@
1
+ require 'benchmark'
2
+ require 'async_enum'
3
+
4
+ sync = Benchmark.measure do
5
+ 5.times{ sleep 0.1 }
6
+ end
7
+
8
+ async = Benchmark.measure do
9
+ 5.times.async.each{ sleep(0.1) }
10
+ end
11
+
12
+ puts sync, async
data/demos/sync.rb ADDED
@@ -0,0 +1,4 @@
1
+ require 'async_enum'
2
+
3
+ enum = 5.times
4
+ puts enum.async.sync == enum
data/lib/async_enum.rb ADDED
@@ -0,0 +1,8 @@
1
+ require 'enumerator/async'
2
+
3
+ module Enumerable
4
+ def async(pool_size = nil)
5
+ enum = self.is_a?(Enumerator) ? self : self.each
6
+ Enumerator::Async.new(enum, pool_size)
7
+ end
8
+ end
@@ -0,0 +1,106 @@
1
+ class Enumerator
2
+ class Async < Enumerator
3
+
4
+ class Lockset
5
+ def initialize
6
+ @semaphores = Hash.new do |locks, key|
7
+ locks[key] = Mutex.new
8
+ end
9
+ end
10
+
11
+ def lock(key = :__default__, &thread_unsafe_block)
12
+ @semaphores[key].synchronize(&thread_unsafe_block)
13
+ end
14
+
15
+ alias_method :evaluate, :instance_exec
16
+ end
17
+
18
+ def initialize(enum, pool_size = nil)
19
+ @enum = enum
20
+ @pool_size = pool_size
21
+ @lockset = Lockset.new
22
+ end
23
+
24
+ def to_a
25
+ @enum.to_a
26
+ end
27
+
28
+ def sync
29
+ @enum
30
+ end
31
+
32
+ alias_method :to_enum, :sync
33
+
34
+ def size
35
+ @enum.size
36
+ end
37
+
38
+ def each(&work)
39
+ raise_error('each') unless block_given?
40
+
41
+ if @pool_size
42
+ threads = @pool_size.times.map do
43
+ Thread.new do
44
+ loop do
45
+ @enum.any? ? evaluate(*@enum.next, &work) : break
46
+ end
47
+ end
48
+ end
49
+ threads.each(&:join)
50
+ @enum.rewind
51
+ else
52
+ unlimited_threads(&work).each(&:join)
53
+ end
54
+ self
55
+ end
56
+
57
+ def with_index(start = 0, &work)
58
+ @enum = @enum.with_index(start)
59
+ if block_given?
60
+ each(&work)
61
+ else
62
+ self
63
+ end
64
+ end
65
+
66
+ def with_object(obj, &work)
67
+ @enum = @enum.with_object(obj)
68
+ if block_given?
69
+ each(&work); obj
70
+ else
71
+ self
72
+ end
73
+ end
74
+
75
+ def map(&work)
76
+ raise_error('map') unless block_given?
77
+
78
+ if @pool_size
79
+ outs = []
80
+ with_index do |item, index|
81
+ outs[index] = evaluate(item, &work)
82
+ end
83
+ outs
84
+ else
85
+ unlimited_threads(&work).map(&:value)
86
+ end
87
+ end
88
+
89
+ private
90
+
91
+ def evaluate(*args, &work)
92
+ @lockset.instance_exec(*args, &work)
93
+ end
94
+
95
+ def unlimited_threads(&work)
96
+ @enum.map do |*args|
97
+ Thread.new{ evaluate(*args, &work) }
98
+ end
99
+ end
100
+
101
+ def raise_error(method)
102
+ raise ArgumentError, "tried to call async #{method} without a block"
103
+ end
104
+
105
+ end
106
+ end
@@ -0,0 +1,12 @@
1
+ require 'test_helper'
2
+
3
+ class BenchmarkTest < Test
4
+
5
+ test 'async is actually async' do
6
+ assert_performance_constant do |n|
7
+ [n, 500].min.times.async.each{ sleep(0.01) }
8
+ end
9
+ end
10
+
11
+
12
+ end
@@ -0,0 +1,41 @@
1
+ require 'test_helper'
2
+
3
+ class DefinitionsTest < Test
4
+
5
+ setup do
6
+ @methods = Enumerator::Async.instance_methods(false)
7
+ end
8
+
9
+ test 'enumerator async exists' do
10
+ assert_equal 'constant', defined?(Enumerator::Async)
11
+ end
12
+
13
+ test 'enumerator async inherits from enumerator' do
14
+ assert_includes Enumerator::Async.ancestors, Enumerator
15
+ end
16
+
17
+ test 'enumerator holds onto enum' do
18
+ enum = Enumerator::Async.new(1..5)
19
+ assert_equal (1..5).to_a, enum.instance_variable_get('@enum').to_a
20
+ end
21
+
22
+ test 'enumerator holds onto pool_size' do
23
+ enum = Enumerator::Async.new(1..5, 5)
24
+ assert_equal 5, enum.instance_variable_get('@pool_size')
25
+ end
26
+
27
+ %w(to_a to_enum sync each map with_index with_object).each do |method|
28
+ test "enumerator async responds to #{method}" do
29
+ assert_includes @methods, method.to_sym
30
+ end
31
+ end
32
+
33
+ test 'enumerable#async exists' do
34
+ assert_includes Enumerable.instance_methods(false), :async
35
+ assert 5.times.respond_to?(:async)
36
+ assert (1..5).respond_to?(:async)
37
+ assert [].respond_to?(:async)
38
+ end
39
+
40
+
41
+ end
@@ -0,0 +1,101 @@
1
+ require 'test_helper'
2
+
3
+ class EnumeratorAsyncTest < Test
4
+
5
+ setup do
6
+ @enum = Enumerator::Async.new( (1..5).each )
7
+ end
8
+
9
+ test 'to_a' do
10
+ assert_equal [1, 2, 3, 4, 5], @enum.to_a
11
+ end
12
+
13
+ test 'sync returns enumerator' do
14
+ refute @enum.sync.is_a?(Enumerator::Async)
15
+ end
16
+
17
+ test 'to_enum alias of sync' do
18
+ assert_equal @enum.sync, @enum.to_enum
19
+ end
20
+
21
+ test 'size' do
22
+ if Enumerator.instance_methods(false).include?(:size)
23
+ assert_equal 5, @enum.size
24
+ else
25
+ assert_raises(NoMethodError) { @enum.size }
26
+ end
27
+ end
28
+
29
+ test 'each with no block' do
30
+ assert_raises(ArgumentError) { @enum.each }
31
+ end
32
+
33
+ test 'each' do
34
+ nums = []
35
+ @enum.each do |i|
36
+ nums << i
37
+ end
38
+ assert_equal @enum.to_a, nums.sort
39
+ end
40
+
41
+ test 'each with splatting' do
42
+ ranges = [ ('a'..'c').to_a ] * 3
43
+ enum = Enumerator::Async.new( ranges.each )
44
+ strs = []
45
+ enum.each do |a, b, c|
46
+ strs << (a + b + c)
47
+ end
48
+ assert_equal %w(abc abc abc), strs
49
+ end
50
+
51
+ test 'with_index no block' do
52
+ pairs = @enum.with_index.to_a[0, 3]
53
+ assert_equal [[1, 0], [2, 1], [3, 2]], pairs
54
+ end
55
+
56
+ test 'with_index' do
57
+ nums = []
58
+ @enum.with_index do |x, i|
59
+ nums << (x - i)
60
+ end
61
+ assert_equal [1, 1, 1, 1, 1], nums
62
+ end
63
+
64
+ test 'with_object no block' do
65
+ h = {}
66
+ pair = @enum.with_object(h).to_a.first
67
+ assert_equal [ 1, h ], pair
68
+ end
69
+
70
+ test 'with_object' do
71
+ to_i = @enum.with_object({}) do |x, hash|
72
+ hash[ x.to_s ] = x
73
+ end
74
+ assert_equal to_i['3'], 3
75
+ end
76
+
77
+ test 'map no block' do
78
+ assert_raises(ArgumentError) { @enum.map }
79
+ end
80
+
81
+ test 'map' do
82
+ squares = @enum.map{ |i| i * i }
83
+ assert_equal [1, 4, 9, 16, 25], squares
84
+ end
85
+
86
+ test 'map keeps order' do
87
+ strs = @enum.map{ |i| sleep rand; i.to_s }
88
+ assert_equal '1 2 3 4 5', strs.join(' ')
89
+ end
90
+
91
+ test 'lock in block' do
92
+ count = 0
93
+ 1000.times.async(5).each do
94
+ lock :count do
95
+ count += 1
96
+ end
97
+ end
98
+ assert_equal 1000, count
99
+ end
100
+
101
+ end
@@ -0,0 +1,24 @@
1
+ require 'test_helper'
2
+
3
+ class ThreadPoolTest < Test
4
+
5
+ test 'pools can speed things up' do
6
+ default_start = Time.now
7
+ 500.times.async.each{ true }
8
+ default_delta = Time.now - default_start
9
+
10
+ rated_start = Time.now
11
+ 500.times.async(10).each{ true }
12
+ rated_delta = Time.now - rated_start
13
+
14
+ assert rated_delta < default_delta
15
+ end
16
+
17
+ test 'pools with ranges' do
18
+ vals = (0..7).async(4).map do |x|
19
+ x + 2
20
+ end
21
+ assert_equal [2, 3, 4, 5, 6, 7, 8, 9], vals
22
+ end
23
+
24
+ end
@@ -0,0 +1,35 @@
1
+ require 'minitest/autorun'
2
+ require 'minitest/benchmark'
3
+ require 'async_enum'
4
+
5
+
6
+
7
+ Test = MiniTest::Unit::TestCase
8
+
9
+
10
+
11
+ def setup(&block)
12
+ define_method('setup', &block)
13
+ end
14
+
15
+ def test(test_name, &block)
16
+ define_method("test_#{ test_name.gsub(/\s+/, '_') }", &block)
17
+ end
18
+
19
+ def cleanup(&block)
20
+ define_method('cleanup', &block)
21
+ end
22
+
23
+ def teardown(&block)
24
+ define_method('teardown', &block)
25
+ end
26
+
27
+
28
+
29
+ def assert_singleton_method(object, sym)
30
+ assert object.respond_to?(sym), "Expected #{object} singleton to respond to #{sym}"
31
+ end
32
+
33
+ def assert_instance_method(object, sym)
34
+ assert object.method_defined?(sym), "Expected #{object} instance to respond to #{sym}"
35
+ end
metadata ADDED
@@ -0,0 +1,93 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: async_enum
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.0.4
5
+ platform: ruby
6
+ authors:
7
+ - aj0strow
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2013-07-19 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: bundler
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - ! '>='
18
+ - !ruby/object:Gem::Version
19
+ version: '0'
20
+ type: :development
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - ! '>='
25
+ - !ruby/object:Gem::Version
26
+ version: '0'
27
+ - !ruby/object:Gem::Dependency
28
+ name: rake
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - ! '>='
32
+ - !ruby/object:Gem::Version
33
+ version: '0'
34
+ type: :development
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - ! '>='
39
+ - !ruby/object:Gem::Version
40
+ version: '0'
41
+ description: iterate over enumerable objects concurrently
42
+ email: alexander.ostrow@gmail.com
43
+ executables: []
44
+ extensions: []
45
+ extra_rdoc_files: []
46
+ files:
47
+ - .gitignore
48
+ - .travis.yml
49
+ - Gemfile
50
+ - README.md
51
+ - Rakefile
52
+ - async_enum.gemspec
53
+ - demos/lock.rb
54
+ - demos/pool_size.rb
55
+ - demos/sleep.rb
56
+ - demos/sync.rb
57
+ - lib/async_enum.rb
58
+ - lib/enumerator/async.rb
59
+ - test/async_enum_test/benchmark_test.rb
60
+ - test/async_enum_test/definitions_test.rb
61
+ - test/async_enum_test/enumerator_async_test.rb
62
+ - test/async_enum_test/thread_pool_test.rb
63
+ - test/test_helper.rb
64
+ homepage: http://github.com/aj0strow/async_enum
65
+ licenses:
66
+ - MIT
67
+ metadata: {}
68
+ post_install_message:
69
+ rdoc_options: []
70
+ require_paths:
71
+ - lib
72
+ required_ruby_version: !ruby/object:Gem::Requirement
73
+ requirements:
74
+ - - ! '>='
75
+ - !ruby/object:Gem::Version
76
+ version: '0'
77
+ required_rubygems_version: !ruby/object:Gem::Requirement
78
+ requirements:
79
+ - - ! '>='
80
+ - !ruby/object:Gem::Version
81
+ version: '0'
82
+ requirements: []
83
+ rubyforge_project:
84
+ rubygems_version: 2.0.5
85
+ signing_key:
86
+ specification_version: 4
87
+ summary: Async Enumerable and Enumerator
88
+ test_files:
89
+ - test/async_enum_test/benchmark_test.rb
90
+ - test/async_enum_test/definitions_test.rb
91
+ - test/async_enum_test/enumerator_async_test.rb
92
+ - test/async_enum_test/thread_pool_test.rb
93
+ - test/test_helper.rb