jruby_threach 0.3.0-java

data/.document

@@ -0,0 +1,5 @@
+lib/**/*.rb
+bin/*
+- 
+features/**/*.feature
+LICENSE.txt

data/.rspec

@@ -0,0 +1 @@
+--color

data/Gemfile

@@ -0,0 +1,14 @@
+source "http://rubygems.org"
+# Add dependencies required to use your gem here.
+# Example:
+#   gem "activesupport", ">= 2.3.5"
+
+# Add dependencies to develop your gem here.
+# Include everything needed to run rake, tests, features, etc.
+group :development do
+  gem "rspec", "~> 2.3.0"
+  gem "yard", "~> 0.6.0"
+  gem "bundler", "~> 1.0.0"
+  gem "jeweler", "~> 1.6.0"
+  gem "rcov", ">= 0"
+end

data/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2011 BillDueber
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.markdown

@@ -0,0 +1,89 @@
+# JRuby_threach
+
+`jruby_threach` adds a method to Enumerable called `#threach` that takes a block and distributes the objects produced by a call to `#each` (or the enumerator of your choice) automatically to a set of consumers.
+
+`jruby_threach` is roughly seven zillion times better than it's vanilla MRI counterpart (`threach`) in that it can handle non-local exits out of the block (`break` or `raise`) without incident. This is possible due to the more powerful queue class available in Java.
+
+## What it does
+
+`jruby_threach` effectively creates a classic producer/consumer setup with a size-restricted queue between them. A single producer (running in the main thread) runs the given iterator (`#each` by default) and stuffs values into the queue. The consumers then grab these values and process them.
+
+A few things can be gleaned from this:
+
+* If your consumer code is faster than your producer code, your consumers are going to be starved
+* You should allocate no more consumers than the speed multiplier between producer and consumer code (e.g., if your producer code -- the call to #each -- is only twice as fast as the block you're passing, don't bother to allocate more than two consumer threads).
+
+## Installation
+
+`jruby_threach` is on rubygems.org, so you should just be able to do
+
+    gem install jruby_threach
+    # or JRuby --1.9 -S gem install JRuby_threach for 1.9
+
+## Use
+
+`jruby_threach` works under JRuby with and without the --1.9 switch. 
+
+    require 'jruby_threach'
+    
+    # Process an array of data with two threads (default is to call #each)    
+    ary.threach(2) {|i| process_data(i)}
+    
+    # Specify the iterator at will
+    ary.threach(2, :each_with_index) {|val, i| process_index_value(i, val)}
+    
+    # #threach doesn't care what the arity of your enumerator is, so long
+    # as the block you pass matches it
+    hash.threach(3) {|k,v| process_key_value(k,v)}
+    
+    # Anything that mixes in enumerable is game. e.g., process lines of a file
+    File.open('myfile.txt').threach(3, :each_line) {|line| process_line(line)}
+    
+    # Pass in a thread count of 0 to just call the underlying method
+    ary.threach(0)
+
+
+## mthreach -- thread the producers as well
+
+There are a couple obvious use cases (e.g., work on a whole list of files) where it might be useful to thread the producer as well. `mthreach` operates on an array of enumerables, using multiple threads to use them to fill a queue simultaneously, and then uses the normal `threach` logic to have multiple consumers pulling from that queue to do work.
+
+    # Use 2 producer threads and 3 consumer threads to process lines
+    Dir.glob('*.txt').map{|f| File.open(f)}.mthreach(2,3,:each_line) {|l|...}
+
+This starts to make more sense if your producer objects aren't super-speedy (e.g., an object that's crawling a list of URLs off the web, hitting a database or set of databases, etc.) or if you have the exemplar case of multiple files (e.g., a set of CSV files where you're going to process each line and stick it into a database). 
+
+
+## Threach::MultiEnum 
+
+You can also use Threach::MultiEnum, the class behind `#mthreach`, by itself, for whenever you have a set of iterators and want to deal with them as if they were a single thing.
+
+    me = Threach::MultiEnum.new(
+       [1..10, 'a'..'z', File.open('myfile.txt')], # array of enumerables
+       :each, # enumerator method to use
+       10,    # size of the internal queue
+       3)     # number of threads. nil => one per enum
+    me.each {|item| process_item(item)}
+
+
+# Things to know
+
+**You're using threads!** Your code needs to be thread-safe (esp. make sure your database connections can cope), and of course we can't guarantee what order the data will be processed in. 
+
+**Spurious warning**  As of this writing, breaking out of a `threach` loop (by calling `break` in the passed block) causes JRuby to print a warning of the form, 'Exception in thread "RubyThread-44: samples.rb:1"org.JRuby.exceptions.JumpException$BreakJump'. Hopefully this can be tracked down and eliminated by the JRuby team.
+
+
+## Contributing to JRuby_threach
+ 
+* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet
+* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it
+* Fork the project
+* Start a feature/bugfix branch
+* Commit and push until you are happy with your contribution
+* Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
+* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
+
+## Copyright
+
+Copyright (c) 2011 BillDueber. See LICENSE.txt for
+further details.
+

data/Rakefile

@@ -0,0 +1,43 @@
+# encoding: utf-8
+
+require 'rubygems'
+require 'bundler'
+begin
+  Bundler.setup(:default, :development)
+rescue Bundler::BundlerError => e
+  $stderr.puts e.message
+  $stderr.puts "Run `bundle install` to install missing gems"
+  exit e.status_code
+end
+require 'rake'
+
+require 'jeweler'
+Jeweler::Tasks.new do |gem|
+  # gem is a Gem::Specification... see http://docs.rubygems.org/read/chapter/20 for more options
+  gem.name = "jruby_threach"
+  gem.homepage = "http://github.com/billdueber/jruby_threach"
+  gem.license = "MIT"
+  gem.summary = %Q{Very simply run a block of code using multiple threads under jruby}
+  gem.description = %Q{Run a block of code in multiple threads. Similar to threach, but with the ability to deal with breaks/exceptions (MRI can't because of unreliable timeouts).}
+  gem.email = "bill@dueber.com"
+  gem.authors = ["BillDueber"]
+  gem.platform = 'java'
+  # dependencies defined in Gemfile
+end
+Jeweler::RubygemsDotOrgTasks.new
+
+require 'rspec/core'
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec) do |spec|
+  spec.pattern = FileList['spec/**/*_spec.rb']
+end
+
+RSpec::Core::RakeTask.new(:rcov) do |spec|
+  spec.pattern = 'spec/**/*_spec.rb'
+  spec.rcov = true
+end
+
+task :default => :spec
+
+require 'yard'
+YARD::Rake::YardocTask.new

data/lib/jruby_threach.rb

@@ -0,0 +1,300 @@
+require 'java'
+java_import java.util.concurrent.ArrayBlockingQueue
+java_import java.util.concurrent.TimeUnit
+java_import org.jruby.exceptions.JumpException::BreakJump
+
+module Threach
+  
+  DEBUG = false
+  
+  # Exception for when a consumer encounters a 'break'
+  class ThreachBreak < RuntimeError; end
+  # Exception to note that another thread has told the rest of us
+  # to wind it down due to an error
+  class ThreachNotMyError < RuntimeError; end
+  # Exception that indicates that we've legitimately run out of data
+  # to process
+  class ThreachEndOfRun < RuntimeError; end
+  
+  
+  # An ArrayBlockingQueue with reasonable defaults
+  # for timeouts.
+  
+  class Queue < ArrayBlockingQueue
+    MS = TimeUnit::MILLISECONDS
+    
+    # Create a new queue 
+    # @param [Integer] size The size of the queue
+    # @param [Integer] timeout_in_ms How long to wait when trying to push or pop
+    # @return [Queue] the new queue
+    def initialize (size=5, timeout_in_ms = 5)
+      super(size)
+      @timeout = timeout_in_ms
+    end
+    
+    # Try to add an object to the queue
+    # @param [Object] obj The object to push
+    # @return [Boolean] true on success, false on timeout
+    def push obj
+      self.offer obj, @timeout, MS
+    end
+    
+    # Pop an object ouf of the queue
+    # @return [Object, nil] nil if it times out; the popped object otherwise
+    def pop
+      self.poll @timeout, MS
+    end
+  end
+
+
+  # A class that encapsulates several enumerables (that respond to the same 
+  # enumerable with the same arity) and allows you to call them as if they
+  # were a single enumerable (using multiple threads to draw from them, if
+  # desired)
+  class MultiEnum
+    include Enumerable
+    
+    # The queue that acts as the common cache for objects pulled
+    # from each of the enumerables
+    attr_accessor :queue
+    
+    # Create a new MultiEnum
+    # @param [Enumerable] enumerators A list of enumerators that you wish to act as a single enum
+    # @param [Symbol] iterator Which iterator to call against each enum
+    # @param [Integer] size The size of the underlying queue
+    # @param [Integer, nil] numthreads The number of threads to dedicate to pulling items 
+    #   off the enumerators and pushing them onto the shared queue. nil or zero implies one for
+    #   each enumerator
+    # @return [Threach::MultiEnum] the new multi-enumerator
+    def initialize enumerators, iterator = :each, size = 5, numthreads=nil
+      @enum = enumerators
+      @iter = iterator
+      @size = size
+      @numthreads = (numthreads.nil? or numthreads == 0) ? enumerators.size : numthreads
+      @queue = Threach::Queue.new(@size)
+    end
+    
+    
+    # Pull records out of the given enumerators using the number of threads
+    # specified at initialization. Order of items is, obviously, not 
+    # guaranteed.
+    #
+    # Also obviously, the passed block need to be of the same arity as the 
+    # enumerator symbol passed into the intializer.
+    # 
+    # An uncaptured exception thrown by any of the enumerators will bring 
+    # the whole thing crashing down. 
+    def each &blk
+      @producers = []
+      tmn = -1
+      @enum.each_slice(@numthreads).each do |eslice|
+        tmn += 1
+        @producers << Thread.new(eslice, tmn) do |eslice, tmn|
+          Thread.current[:threach_multi_num] = "p#{tmn}"
+          begin
+            eslice.size.times do |i|
+              eslice[i].send(@iter) do |*x|
+                # puts "...pushing #{x}"
+                @queue.put "#{Thread.current[:threach_multi_num]}: #{x}"
+              end
+            end
+            @queue.put :threach_multi_eof
+          rescue Exception => e
+            @queue.put :threach_multi_eof
+            raise StopIteration.new "Error in #{eslice.inspect}: #{e.inspect}"
+          end
+        end
+      end
+
+      done = 0
+      
+      while done < @numthreads
+        d = @queue.take
+        # puts "...pulling #{d}"
+        if d == :threach_multi_eof
+          done += 1 
+          next
+        end
+        yield d
+      end
+      
+      @producers.each {|p| p.join}
+    end
+  end
+  
+end
+
+# Enumerable is monkey-patched to provide two new methods: #threach and 
+# #mthreach. 
+module Enumerable
+  
+  # Build up a MultiEnum from the calling object and run threach against
+  # it
+  # @param [Integer, nil] pthreads The number of producer threads to run within the 
+  #   created Threach::MultiEnum
+  # @param [Integer] threads The number of consumer threads to run in #threach
+  # @param [Symbol] iterator Which iterator to call (:each, :each_with_index, etc.)
+  # 
+  # @example
+  #   [1..10, 'a'..'z'].mthreach(2,2) {|i| process_item(i)}
+  # 
+  def mthreach(pthreads=nil, threads = 0, iterator = :each,  &blk)
+    me = Threach::MultiEnum.new(self, iterator, threads * 3, pthreads)
+    me.send(:threach, threads, iterator, &blk)
+  end
+  
+  # Run the passed block using the given iterator using the given 
+  # number of threads. If one of the consumer threads bails for any reason
+  # (break, throw an un-rescued error), the whole thing will shut down in an
+  # orderly fashion.
+  # @param [Integer] threads How many threads to use. 0 means to skip the whole 
+  #   threading thing completely and just directly call the indicated iterator
+  # @param [Symbol] iterator Which iterator to use (:each, :each_with_index, :each_line, 
+  #   etc.). 
+  def threach(threads = 0, iterator = :each, &blk)
+    
+    # With no extra threads, just spin up the passed iterator
+    if threads == 0
+      self.send(iterator, &blk)
+    else
+      # Get a java BlockingQueue for the producer to dump stuff into
+      bq = Threach::Queue.new(threads * 2) # capacity is twice the number of threads
+      
+      # And another to store errors
+      errorq = Threach::Queue.new(threads + 1)
+      
+      # A boolean to let us know if things are going wonky
+      bail = false
+      outofdata = false
+      
+      # Build up a set of consumers
+      
+      consumers = []
+      threads.times do |i|
+        consumers << Thread.new(i) do |i|
+          Thread.current[:threach_num] = i
+          begin
+            while true
+              obj = bq.pop
+
+              # Should we be bailing?
+              if bail
+                print "Thread #{Thread.current[:threach_num]}: BAIL!\n" if Threach::DEBUG
+                Thread.current[:threach_bail] = true
+                raise Threach::ThreachNotMyError.new, "bailing", nil
+              end
+              
+            
+              # If the return value is nil, it timed out. See if there's
+              # anything wrong, or if we've run out of work
+              if obj.nil?
+                if outofdata
+                  Thread.current[:threach_outofdata] = true
+                  raise Threach::ThreachEndOfRun.new, "out of work", nil
+                end
+                # otherwise, try to pop again
+                next 
+              end
+              
+              # Otherwise, do the work
+              blk.call(*obj)
+            end
+          
+          rescue Threach::ThreachNotMyError => e
+            print "Thread #{Thread.current[:threach_num]}: Not my error\n" if Threach::DEBUG
+            Thread.current[:threach_bail] = true            
+            # do nothing; wasn't my error, so I just bailed
+          
+          rescue Threach::ThreachEndOfRun => e
+            print "Thread #{Thread.current[:threach_num]}: End of run\n" if Threach::DEBUG
+            Thread.current[:threach_bail] = true            
+            # do nothing; everything exited normally 
+            
+          rescue Exception => e
+            print "Thread #{Thread.current[:threach_num]}: Exception #{e.inspect}: #{e.message}\n" if Threach::DEBUG
+            # Some other error; let everyone else know
+            bail = true
+            Thread.current[:threach_bail]
+            errorq.push e
+          ensure
+            # OK, I don't understand this, but I'm unable to catch org.jruby.exceptions.JumpException$BreakJump
+            # But if I get here and nothing else is set, that means I broke and need to deal with
+            # it accordingly
+            unless Thread.current[:threach_bail] or Thread.current[:threach_outofdata]
+              print "Thread #{Thread.current[:threach_num]}: broke out of loop\n" if Threach::DEBUG
+              bail = true
+            end
+          end
+        end
+      end
+      
+      
+      # Now, our producer
+      
+      # Start running the given iterator and try to push stuff
+      
+      begin
+        self.send(iterator) do |*x|
+          until successful_push = bq.push(x)
+            # if we're in here, we got a timeout. Check for errors
+            raise Threach::ThreachNotMyError.new, "bailing", nil if bail if bail
+          end
+          print "Queued #{x}\n" if Threach::DEBUG
+        end
+
+        # We're all done. Let 'em know
+        print "Setting outofdata to true\n" if Threach::DEBUG
+        outofdata = true
+      
+      rescue NativeException => e
+        print "Producer rescuing native exception #{e.inspect}" if Threach::DEBUG
+      
+      rescue Threach::ThreachNotMyError => e
+        print "Producer: not my error\n" if Threach::DEBUG
+        # do nothing. Not my error
+        
+      rescue Exception => e
+        print "Producer: exception\n" if Threach::DEBUG
+        bail = true
+        errorq.push e
+      end
+      
+      # Finally, #join the consumers
+      
+      consumers.each {|t| t.join}
+      
+      # Everything's done. If there's an error on the stack, raise it
+       if e = errorq.peek
+         print "Producer: raising #{e.inspect}\n" if Threach::DEBUG
+         raise e, e.message, nil
+       end
+      
+
+    end
+  end
+end
+
+__END__
+
+class DelayedEnum
+  include Enumerable
+  
+  def initialize coll
+    @coll = coll
+  end
+  
+  def each &blk
+    @coll.each do |i|
+      sleep 0.1
+      yield i
+    end
+  end
+end
+
+c = DelayedEnum.new((1..10).to_a)
+d = DelayedEnum.new(('A'..'N').to_a)
+e = DelayedEnum.new((20..30).to_a)
+f = DelayedEnum.new(('m'..'z').to_a)
+
+[c,d,e,f].mthreach(2,2) {|i| print "#{Thread.current[:threach_num]}: #{i}\n"}
+  

data/samples.rb

@@ -0,0 +1,60 @@
+load 'lib/jruby_threach.rb'
+
+# Simple: just do it
+puts "\n\nProblem-free execution\n"
+(1..10).threach(3) do |i|
+  print "#{Thread.current[:threach_num]}: #{i}\n"
+  (Thread.current[:threach_num] % 3) == 1 ? sleep(0.1): sleep(0.3)
+end
+
+# Break out of it. If any one thread breaks, they all grind 
+# to a half
+
+puts "\n\nBreak out of the block\n"
+(1..10).threach(3) do |i|
+  print "#{Thread.current[:threach_num]}: #{i}\n"
+  break if i == 5
+  (Thread.current[:threach_num] % 3) == 1 ? sleep(0.1): sleep(0.3)
+end
+
+# We also capture errors that aren't handled in the block, stop
+# everything, and then re-raise the error to the calling code
+
+puts "\n\nStop execution on error and pass error to calling code\n"
+begin
+  (1..10).threach(3) do |i|
+    print "#{Thread.current[:threach_num]}: #{i}\n"
+    raise RuntimeError.new, "oops", nil if i == 5
+    (Thread.current[:threach_num] % 3) == 1 ? sleep(0.1): sleep(0.3)
+  end
+rescue RuntimeError => e
+  print "Caught an error that stopped execution\n"
+end
+
+# Of course, if you deal with the error within the block,
+# there's no problem.
+
+puts "\n\nRescue error within passed block\n"
+begin
+  (1..10).threach(3) do |i|
+    begin
+      print "#{Thread.current[:threach_num]}: #{i}\n"
+      raise RuntimeError.new, "oops", nil if i == 5
+      (Thread.current[:threach_num] % 3) == 1 ? sleep(0.1): sleep(0.3)
+    rescue RuntimeError => e
+      print "Caught error inside block; just ignore it and move on\n"
+    end
+  end
+rescue RuntimeError => e
+  print "Caught an error that stopped execution\n" # won't ever run
+end
+
+# Nesting example 
+files = ['Rakefile', 'README.rdoc', 'LICENSE.txt']
+files.threach(3) do |f|
+  File.open(f).threach(2, :each_line) do |line|
+    print "#{f}: #{line.chomp[0..20]}\n"
+  end
+end
+
+

data/spec/jruby_threach_spec.rb

@@ -0,0 +1,7 @@
+require File.expand_path(File.dirname(__FILE__) + '/spec_helper')
+
+describe "JrubyThreach" do
+  it "fails" do
+    fail "hey buddy, you should probably rename this file and start specing for real"
+  end
+end

data/spec/spec_helper.rb

@@ -0,0 +1,12 @@
+$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
+$LOAD_PATH.unshift(File.dirname(__FILE__))
+require 'rspec'
+require 'jruby_threach'
+
+# Requires supporting files with custom matchers and macros, etc,
+# in ./support/ and its subdirectories.
+Dir["#{File.dirname(__FILE__)}/support/**/*.rb"].each {|f| require f}
+
+RSpec.configure do |config|
+  
+end

metadata

@@ -0,0 +1,123 @@
+--- !ruby/object:Gem::Specification 
+name: jruby_threach
+version: !ruby/object:Gem::Version 
+  prerelease: 
+  version: 0.3.0
+platform: java
+authors: 
+- BillDueber
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2011-06-30 00:00:00 -07:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: rspec
+  requirement: &id001 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        version: 2.3.0
+  type: :development
+  prerelease: false
+  version_requirements: *id001
+- !ruby/object:Gem::Dependency 
+  name: yard
+  requirement: &id002 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        version: 0.6.0
+  type: :development
+  prerelease: false
+  version_requirements: *id002
+- !ruby/object:Gem::Dependency 
+  name: bundler
+  requirement: &id003 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        version: 1.0.0
+  type: :development
+  prerelease: false
+  version_requirements: *id003
+- !ruby/object:Gem::Dependency 
+  name: jeweler
+  requirement: &id004 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        version: 1.6.0
+  type: :development
+  prerelease: false
+  version_requirements: *id004
+- !ruby/object:Gem::Dependency 
+  name: rcov
+  requirement: &id005 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: "0"
+  type: :development
+  prerelease: false
+  version_requirements: *id005
+description: Run a block of code in multiple threads. Similar to threach, but with the ability to deal with breaks/exceptions (MRI can't because of unreliable timeouts).
+email: bill@dueber.com
+executables: []
+
+extensions: []
+
+extra_rdoc_files: 
+- LICENSE.txt
+- README.markdown
+files: 
+- .document
+- .rspec
+- Gemfile
+- LICENSE.txt
+- README.markdown
+- Rakefile
+- lib/jruby_threach.rb
+- samples.rb
+- spec/jruby_threach_spec.rb
+- spec/spec_helper.rb
+has_rdoc: true
+homepage: http://github.com/billdueber/jruby_threach
+licenses: 
+- MIT
+post_install_message: 
+rdoc_options: []
+
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: -1042269137
+      segments: 
+      - 0
+      version: "0"
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.6.2
+signing_key: 
+specification_version: 3
+summary: Very simply run a block of code using multiple threads under jruby
+test_files: []
+