xpool 0.9.0.2 → 0.10.0

data/.travis.yml

@@ -1,5 +1,5 @@
 rvm:
-  - 1.9.2
+#  - 1.9.2
   - 1.9.3
   - 2.0.0
   - ruby-head

data/.yardopts

@@ -0,0 +1,5 @@
+-m markdown -M redcarpet
+-
+README.md
+LICENSE.txt
+ChangeLog.txt

data/ChangeLog.txt

@@ -1,6 +1,52 @@
+== v0.10.0
+	- default to two subprocesses if CPU core count cannot be guessed.
+	Which is change from the old default of five.
+
+	- add XPool#resize
+	Resize the pool at runtime.
+	If subprocesses are removed from the pool a graceful shutdown is performed on 
+	each subprocess that is removed (unlike resize!, which forces a shutdown).
+
+	- add XPool#expand
+	Expand the pool with X subprocesses, where X is the argument to expand(…).
+
+	- add XPool#shrink
+	Shrink the pool by X subprocesses, where X is the argument to shrink(…).
+	A graceful shutdown is performed on any subprocesses removed from the pool.
+
+	- add XPool#shrink!
+	Shrink the pool by X subprocesses, where X is the argument to shrink!(…)
+	A forceful shutdown is performed on any subprocesses removed from the pool.
+
+	- XPool#resize! now accepts a Fixnum, not a Range.
+	If you want to resize the pool to two subprocesses, the API is now:
+	pool.resize!(2)
+	
+	- XPool#failed_processes persists after a shutdown
+	If a process in the pool has failed and the pool is subsequently shutdown the
+	failed process will remain in XPool#failed_processes. When a failed process
+	is restarted the pool is repopulated with an active subprocess and can be 
+	used to schedule work again.
+	
+	- optimize XPool#resize! 
+	A few optimizations:
+	
+	* when new_size == current_size, there's no resize. nothing happens. it's 
+	a no-op. 
+
+	* when new_size > current_size, only X new subprocesses are spawned, where X is 
+	the difference between "new_size - current_size". 
+
+	* when new_size < current_size, all subprocesses indexed after new_size are
+	shutdown and removed from the pool. no new subprocesses are spawned.
+
+	This new behavior reuses subprocesses that are already active in the pool when
+	it can, where as before we always spawned new subprocesses, no matter what the
+	case.
+
 == v0.9.0.1,v0.9.0.2
 	- doc improvements
-	revised & improved the README & API documentation.
+	Revised & improved the README & API documentation.
 
 == v0.9.0
 	- upgrade to ichannel v5.1.1

data/Gemfile

@@ -3,5 +3,6 @@ group :development do
   gem 'rake'
   gem 'pry'
   gem 'ruby-prof'
+  gem 'mocha'
 end
 gemspec

data/README.md

@@ -10,37 +10,50 @@ __OVERVIEW__
 
 __DESCRIPTION__
 
-xpool is a lightweight process pool. The pool manages a group of subprocesses
-that are used when the pool is asked to dispatch a 'unit of work'. A 
-'unit of work' is defined as any object that implements the `run` method.
-
-All subprocesses in the pool have their own message queue that the pool places
-work onto according to a very simple algorithm: the subprocess who has scheduled
-the least amount of work is the subprocess who is asked to put the work on its
-queue. This helps ensure an even distribution of work among all subprocesses in 
-the pool. The message queue that each subprocess has is also what ensures 
-work can be queued when the pool becomes dry (all subprocesses are busy). 
-
-Incase a unit of work raises an exception that it does not handle xpool will
-rescue the exception and mark the process as 'failed'. A failed process can be
-restarted, and it is also possible to access the backtrace of a failed process 
-through `XPool` and `XPool::Process` objects. The exception is also re-raised 
-so that you can see a process has failed from the output ruby prints when an 
-exception is left unhandled.
-
-__POOL SIZE__
+xpool is a lightweight process pool. A pool manages a group of subprocesses
+that are used when it is asked to dispatch a 'unit of work'. A 'unit of work' 
+is defined as any object that implements the `run` method.
+
+In order to send a 'unit of work' between processes each subprocess has its own
+'message queue' that the pool writes to when it has been asked to schedule a 
+unit of work. A unit of work is serialized(on write to queue), and 
+deserialized(on read from queue). The serializer used under the hood is called 
+[Marshal](http://rubydoc.info/stdlib/core/Marshal) and might be familiar to 
+you already.
+
+The logic for scheduling a unit of work is straightforward. A pool asks each 
+and every subprocess under its control how frequently its message queue has 
+been written to. The subprocess with the queue that has the least writes is told
+to schedule the next unit of work. In practical terms this means if you have a 
+pool with five subprocesses and schedule a unit of work five times, each 
+subprocess in the pool would have executed the unit of work once.
+
+A pool can become "dry" whenever all its subprocesses are busy. If you schedule
+a unit of work on a dry pool the same scheduling logic apllies but instead of
+the unit of work executing right away it will be executed whenever the 
+assigned subprocess is no longer busy. It is also possible to query the pool 
+and ask if it is dry, but you can also ask an individual subprocess if it is
+busy.
 
 By default xpool will create a pool with X subprocesses, where X is the number 
 of cores on your CPU. This seems like a reasonable default, but if you should 
 decide to choose otherwise you can set the size of the pool when it is 
-initialized. The pool can also be resized at runtime if you decide you need to 
-scale up or down.
+initialized. The pool can also be resized at runtime if you decide you need 
+to scale up or down.
+
+A unit of work may fail whenever an exception is left unhandled. When this 
+happens xpool rescues the exception, marks the process as "failed", and 
+re-raises the exception so that the failure can be seen. Finally, the process 
+running the unit of work exits, and pool is down one process. A failed process 
+can be restarted and interacted with, though, so it is possible to recover.
 
 __EXAMPLES__
 
 The examples don't demonstrate everything that xpool can do. The 
-[API docs](http://rubydoc.info/github/robgleeson/xpool) 
-cover the missing pieces.
+[API docs](http://rubydoc.info/github/robgleeson/xpool)
+and 
+[docs/](https://github.com/robgleeson/xpool/tree/master/docs)
+directory cover the missing pieces.
 
 _1._
 

data/docs/unhandled_exceptions.md

@@ -0,0 +1,43 @@
+__ABSTRACT__
+
+A unit of work may fail whenever an exception is left unhandled. When this 
+happens xpool rescues the exception, marks the process as "failed", and 
+re-raises the exception so that the failure can be seen. Finally, the 
+process running the unit of work exits, and pool is down one process. A failed 
+process can be restarted and interacted with, though, so it is possible to 
+recover.
+
+__DETAILS__
+
+A quick summary of the classes and methods that are most interesting when
+interacting with failed processes:
+
+  - [XPool#failed_processes](http://rubydoc.info/github/robgleeson/xpool/XPool#failed_processes-instance_method)  
+  Returns an array of 
+  [XPool::Process](http://rubydoc.info/github/robgleeson/xpool/XPool/Process)
+  objects that are in a failed state.
+
+  - [XPool::Process](http://rubydoc.info/github/robgleeson/xpool/XPool/Process)  
+  Provides an object oriented interface on top of a subprocess in the pool.   
+  The most interesting methods when a process is in a 'failed' state might be 
+  [XPool::Process#restart](http://rubydoc.info/github/robgleeson/xpool/XPool/Process#restart-instance_method) 
+  and
+  [XPool::Process#backtrace](http://rubydoc.info/github/robgleeson/xpool/XPool/Process#backtrace-instance_method).
+
+__EXAMPLES__
+
+__1.__
+
+```ruby
+class Unit
+  def run
+    raise RuntimeError, "", []
+  end
+end
+
+pool = XPool.new 2
+pool.schedule Unit.new
+sleep 0.05
+pool.failed_processes.each(&:restart)
+pool.shutdown
+```

data/lib/xpool.rb

@@ -41,6 +41,55 @@ class XPool
     @pool = Array.new(size) { Process.new }
   end
 
+  #
+  # @param [Fixnum] number
+  #   The number of subprocesses to add to the pool.
+  #
+  # @return
+  #   (see XPool#resize!)
+  #
+  def expand(number)
+    resize! size + number
+  end
+
+  #
+  # @param [Fixnum] number
+  #   The number of subprocesses to remove from the pool.
+  #   A graceful shutdown is performed.
+  #
+  # @raise
+  #   (see XPool#shrink!)
+  #
+  # @return
+  #   (see Xpool#shrink!)
+  #
+  def shrink(number)
+    present_size = size
+    raise_if number > present_size,
+      ArgumentError,
+      "cannot shrink pool by #{number}. pool is only #{present_size} in size."
+    resize present_size - number
+  end
+
+  #
+  # @param [Fixnum] number
+  #   The number of subprocesses to remove from the pool.
+  #   A forceful shutdown is performed.
+  #
+  # @raise [ArgumentError]
+  #   When _number_ is greater than {#size}.
+  #
+  # @return
+  #   (see XPool#resize!)
+  #
+  def shrink!(number)
+    present_size = size
+    raise_if number > present_size,
+      ArgumentError,
+      "cannot shrink pool by #{number}. pool is only #{present_size} in size."
+    resize! present_size - number
+  end
+
   #
   # @return [Array<XPool::Process>]
   #   Returns an Array of failed processes.
@@ -104,23 +153,32 @@ class XPool
   end
 
   #
-  # Resize the pool.
-  # All subprocesses in the pool are abruptly stopped through {#shutdown!} and
-  # a new pool the size of _range_ is created.
+  # Resize the pool (gracefully, if neccesary)
+  #
+  # @param
+  #   (see XPool#resize!)
+  #
+  # @return [void]
+  #
+  def resize(new_size)
+    _resize new_size, false
+  end
+
+  #
+  # Resize the pool (with force, if neccesary).
   #
   # @example
   #   pool = XPool.new 5
-  #   pool.resize! 1..3
+  #   pool.resize! 3
   #   pool.shutdown
   #
-  # @param [Range] range
+  # @param [Fixnum] new_size
   #   The new size of the pool.
   #
   # @return [void]
   #
-  def resize!(range)
-    shutdown!
-    @pool = range.to_a.map { Process.new }
+  def resize!(new_size)
+    _resize new_size, true
   end
 
   #
@@ -149,9 +207,7 @@ class XPool
   #   Returns the number of alive subprocesses in the pool.
   #
   def size
-    @pool.count do |process|
-      process.alive?
-    end
+    @pool.count(&:alive?)
   end
 
   #
@@ -162,9 +218,13 @@ class XPool
     @pool.all?(&:busy?)
   end
 
-  #
-  # Count the number of CPU cores available.
-  #
+private
+  def raise_if(predicate, e, m)
+    if predicate
+      raise e, m
+    end
+  end
+
   def number_of_cpu_cores
     case RbConfig::CONFIG['host_os']
     when /linux/
@@ -174,7 +234,26 @@ class XPool
     when /solaris/
       Integer(`kstat -m cpu_info | grep -w core_id | uniq | wc -l`)
     else
-      5
+      2
+    end
+  end
+
+  def _resize(new_size, with_force)
+    if Range === new_size
+      warn "[DEPRECATED] XPool#resize! no longer accepts a Range." \
+           "Please use a Fixnum instead."
+      new_size = range.to_a.size
+    end
+    new_size -= 1
+    old_size = size - 1
+    if new_size == old_size
+      # do nothing
+    elsif new_size < old_size
+      meth = with_force ? :shutdown! : :shutdown
+      @pool[new_size+1..old_size].each(&meth)
+      @pool = @pool[0..new_size]
+    else
+      @pool += Array.new(new_size - old_size) { Process.new }
     end
   end
 end

data/lib/xpool/process.rb

@@ -130,7 +130,7 @@ private
     Process.wait @id
   rescue Errno::ECHILD, Errno::ESRCH
   ensure
-    @states = {dead: true}
+    @states = {dead: true} unless failed?
     @shutdown = true
     @channel.close
     @s_channel.close

data/lib/xpool/version.rb

@@ -1,3 +1,3 @@
 class XPool
-  VERSION = "0.9.0.2"
+  VERSION = "0.10.0"
 end

data/test/setup.rb

@@ -2,6 +2,7 @@ require 'bundler/setup'
 require 'xpool'
 require 'test/unit'
 require 'fileutils'
+require 'mocha/setup'
 require_relative 'support/sleeper'
 require_relative 'support/io_writer'
 require_relative 'support/raiser'

data/test/xpool_test.rb

@@ -1,10 +1,12 @@
 require_relative 'setup'
 class XPoolTest < Test::Unit::TestCase
+  POOL_SIZE = 2
   def setup
-    @pool = XPool.new 5
+    @pool = XPool.new POOL_SIZE
   end
 
   def teardown
+    mocha_teardown
     @pool.shutdown!
   end
 
@@ -14,38 +16,38 @@ class XPoolTest < Test::Unit::TestCase
   end
 
   def test_size_with_graceful_shutdown
-    assert_equal 5, @pool.size
+    assert_equal POOL_SIZE, @pool.size
     @pool.shutdown
     assert_equal 0, @pool.size
   end
 
   def test_size_with_forceful_shutdown
-    assert_equal 5, @pool.size
+    assert_equal POOL_SIZE, @pool.size
     @pool.shutdown!
     assert_equal 0, @pool.size
   end
 
   def test_queue
-    @pool.resize! 1..1
-    writers = Array.new(5) { IOWriter.new }
+    @pool.resize! 1
+    writers = Array.new(POOL_SIZE) { IOWriter.new }
     writers.each { |writer| @pool.schedule writer }
     @pool.shutdown
     writers.each { |writer| assert writer.wrote_to_disk? }
   end
 
   def test_distribution_of_work
-    subprocesses = (0..4).map { @pool.schedule Sleeper.new(0.1) }
+    subprocesses = Array.new(POOL_SIZE) { @pool.schedule Sleeper.new(0.1) }
     subprocesses.each { |subprocess| assert_equal 1, subprocess.frequency }
   end
 
   def test_resize!
-    @pool.resize! 1..1
+    @pool.resize! 1
     assert_equal 1, @pool.instance_variable_get(:@pool).size
   end
 
   def test_dry?
     refute @pool.dry?
-    5.times { @pool.schedule Sleeper.new(0.5) }
+    POOL_SIZE.times { @pool.schedule Sleeper.new(0.5) }
     sleep 0.1
     assert @pool.dry?
   end
@@ -54,6 +56,42 @@ class XPoolTest < Test::Unit::TestCase
     @pool.schedule Raiser.new
     sleep 0.1
     assert_equal 1, @pool.failed_processes.size
-    assert_equal 4, @pool.size
+    assert_equal POOL_SIZE - 1, @pool.size
+  end
+
+  def test_failed_processes_after_shutdown
+    @pool.schedule Raiser.new
+    @pool.shutdown
+    refute @pool.failed_processes.empty?
+  end
+
+  def test_failed_process_to_repopulate_pool
+    @pool.schedule Raiser.new
+    @pool.shutdown
+    @pool.failed_processes.each(&:restart)
+    assert_equal 1, @pool.size
+  end
+
+  def test_expand
+    @pool.expand 1
+    assert_equal POOL_SIZE + 1, @pool.size
+  end
+
+  def test_shrink
+    XPool::Process.any_instance.expects(:shutdown).once
+    @pool.shrink 1
+    assert_equal POOL_SIZE - 1, @pool.size
+  end
+
+  def test_shrink!
+    XPool::Process.any_instance.expects(:shutdown!).once
+    @pool.shrink! 1
+    assert_equal POOL_SIZE - 1, @pool.size
+  end
+
+  def test_shrink_with_excess_number
+    assert_raises ArgumentError do
+      @pool.shrink! POOL_SIZE + 1
+    end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: xpool
 version: !ruby/object:Gem::Version
-  version: 0.9.0.2
+  version: 0.10.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-03-22 00:00:00.000000000 Z
+date: 2013-03-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: ichannel
@@ -37,6 +37,7 @@ files:
 - .gitignore
 - .pryrc
 - .travis.yml
+- .yardopts
 - ChangeLog.txt
 - Gemfile
 - LICENSE.txt
@@ -44,6 +45,7 @@ files:
 - Rakefile
 - bench/.gitkeep
 - bench/pool-schedule.rb
+- docs/unhandled_exceptions.md
 - lib/xpool.rb
 - lib/xpool/process.rb
 - lib/xpool/version.rb
@@ -68,7 +70,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -610761221262095322
+      hash: -4088700810187342227
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -77,7 +79,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -610761221262095322
+      hash: -4088700810187342227
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.23