workers 0.0.9 → 0.1.0

data/README.md

@@ -2,6 +2,7 @@
 
 Workers is a Ruby gem for performing work in background threads.
 Design goals include high performance, low latency, simple API, customizability, and multi-layered architecture.
+It provides a number of simple to use classes that solve a wide range of concurrency problems.
 It is used by [Tribe](https://github.com/chadrem/tribe "Tribe") to implement event-driven actors.
 
 ## Installation
@@ -74,53 +75,31 @@ The Worker class is designed to be customized through inheritence and its event
 Note that you can use custom workers without a pool.
 This effectively gives you direct access to a single event-driven thread.
 
-## Timers
-
-Timers provide a way to execute code in the future:
-
-    # Create a one shot timer that executes in 1.5 seconds.
-    timer = Workers::Timer.new(1.5) do
-      puts 'Hello world'
-    end
-
-    # Create a periodic timer that loops infinitely or until 'cancel' is called.
-    timer = Workers::PeriodicTimer.new(1) do
-      puts 'Hello world many times'
-    end
+## Pools - Advanced
 
-    # Let the timer print some lines.
-    sleep(5)
-
-    # Shutdown the timer.
-    timer.cancel
-
-## Schedulers
+As shown above, pools effectively allow a group of workers to share a work queue.
+They have a few additional methods described below:
 
-Schedulers are what trigger Timers to fire.
-The system has a global default scheduler which should meet most needs (Workers.scheduler).
-You can create additional or custom ones as necessary:
-
-    # Create a workers pool with a larger than default thread count (optional).
-    pool = Workers::Pool.new(:size => 100)
+    # Create a pool.
+    pool = Workers::Pool.new
 
-    # Create a scheduler.
-    scheduler = Workers::Scheduler.new(:pool => pool)
+    # Return the number of workers in the pool.
+    pool.size
 
-    # Create a timer that uses the above scheduler.
-    timer = Workers::Timer.new(1, :scheduler => scheduler) do
-      puts 'Hello world'
-    end
+    # Increase the number of workers in the pool.
+    pool.expand(5)
 
-    # Wait for the timer to fire.
-    sleep(5)
+    # Decrease the number of workers in the pool.
+    pool.contract(5)
 
-    # Shutdown the scheduler.
-    scheduler.dispose
+    # Resize the pool size to a specific value.
+    pool.resize(20)
 
-### Tasks
+## Tasks
 
 Tasks and task groups build on top of worker pools.
-They provide a means of parallelizing expensive computations and collecing the results:
+They provide a means of parallelizing expensive computations and collecing the results.
+These are the classes you normally work with in your application level code.
 
     # Create a task group (it contains a pool of workers).
     group = Workers::TaskGroup.new
@@ -156,7 +135,7 @@ They provide a means of parallelizing expensive computations and collecing the r
       t.exception  # The exception if one exists.
     end
 
-### Parallel Map
+## Parallel Map
 
 Task groups and tasks are the building blocks of parallel map:
 
@@ -169,6 +148,49 @@ The below syntax is equivalent to the above:
 
 Note that any failures will cause an exception to be raised.
 
+## Timers
+
+Timers provide a way to execute code in the future:
+
+    # Create a one shot timer that executes in 1.5 seconds.
+    timer = Workers::Timer.new(1.5) do
+      puts 'Hello world'
+    end
+
+    # Create a periodic timer that loops infinitely or until 'cancel' is called.
+    timer = Workers::PeriodicTimer.new(1) do
+      puts 'Hello world many times'
+    end
+
+    # Let the timer print some lines.
+    sleep(5)
+
+    # Shutdown the timer.
+    timer.cancel
+
+## Schedulers
+
+Schedulers are what trigger Timers to fire.
+The system has a global default scheduler which should meet most needs (Workers.scheduler).
+You can create additional or custom ones as necessary:
+
+    # Create a workers pool with a larger than default thread count (optional).
+    pool = Workers::Pool.new(:size => 100)
+
+    # Create a scheduler.
+    scheduler = Workers::Scheduler.new(:pool => pool)
+
+    # Create a timer that uses the above scheduler.
+    timer = Workers::Timer.new(1, :scheduler => scheduler) do
+      puts 'Hello world'
+    end
+
+    # Wait for the timer to fire.
+    sleep(5)
+
+    # Shutdown the scheduler.
+    scheduler.dispose
+
 ## Options (defaults below):
 
     pool = Workers::Pool.new(
@@ -189,17 +211,6 @@ Note that any failures will cause an exception to be raised.
       :callback => nil                  # The proc to execute (provide this or a block, but not both).
     )
 
-    timer = Workers::PeriodicTimer.new(1,
-      :logger => nil,                   # Ruby logger instance.
-      :scheduler => Workers.scheduler,  # The scheduler that manages execution.
-      :callback => nil                  # The proc to execute (provide this or a block, but not both).
-    )
-
-    scheduler = Workers::Scheduler.new(
-      :logger => nil,                   # Ruby logger instance.
-      :pool => Workers::Pool.new        # The workers pool used to execute timer callbacks.
-    )
-
     group = Workers::TaskGroup.new(
       :logger => nil,                   # Ruby logger instance.
       :pool => Workers::Pool.new        # The workers pool used to execute timer callbacks.
@@ -212,6 +223,17 @@ Note that any failures will cause an exception to be raised.
       :finished => nil,                 # Callback to execute after attempting to run the task.
     )
 
+    timer = Workers::PeriodicTimer.new(1,
+      :logger => nil,                   # Ruby logger instance.
+      :scheduler => Workers.scheduler,  # The scheduler that manages execution.
+      :callback => nil                  # The proc to execute (provide this or a block, but not both).
+    )
+
+    scheduler = Workers::Scheduler.new(
+      :logger => nil,                   # Ruby logger instance.
+      :pool => Workers::Pool.new        # The workers pool used to execute timer callbacks.
+    )
+
 ## Contributing
 
 1. Fork it

data/lib/workers/event.rb

@@ -6,6 +6,8 @@ module Workers
     def initialize(command, data)
       @command = command
       @data = data
+
+      return nil
     end
   end
 end

data/lib/workers/log_proxy.rb

@@ -4,22 +4,32 @@ module Workers
 
     def initialize(logger)
       @logger = logger.is_a?(Workers::LogProxy) ? logger.logger : logger
+
+      return nil
     end
 
     def debug(msg)
       @logger.debug(msg) if @logger
+
+      return nil
     end
 
     def info(msg)
       @logger.info(msg) if @logger
+
+      return nil
     end
 
     def warn(msg)
       @logger.warn(msg) if @logger
+
+      return nil
     end
 
     def error(msg)
       @logger.error(msg) if @logger
+
+      return nil
     end
   end
 end

data/lib/workers/periodic_timer.rb

@@ -4,6 +4,8 @@ module Workers
       options[:repeat] = true
 
       super(delay, options, &block)
+
+      return nil
     end
   end
 end

data/lib/workers/pool.rb

@@ -6,12 +6,15 @@ module Workers
 
     def initialize(options = {})
       @logger = Workers::LogProxy.new(options[:logger])
-      @size = options[:size] || Workers::Pool::DEFAULT_POOL_SIZE
       @worker_class = options[:worker_class] || Workers::Worker
-
       @input_queue = Queue.new
-      @workers = []
-      @size.times { @workers << @worker_class.new(:input_queue => @input_queue) }
+      @lock = Monitor.new
+      @workers = Set.new
+      @size = 0
+
+      expand(options[:size] || Workers::Pool::DEFAULT_POOL_SIZE)
+
+      return nil
     end
 
     def enqueue(command, data = nil)
@@ -27,24 +30,91 @@ module Workers
     end
 
     def shutdown(&block)
-      @size.times { enqueue(:shutdown, block) }
+      @lock.synchronize do
+        @size.times do
+          enqueue(:shutdown, block)
+        end
+      end
 
       return nil
     end
 
     def join(max_wait = nil)
-      return @workers.map { |w| w.join(max_wait) }
+      results = @workers.map { |w| w.join(max_wait) }
+      @workers.clear
+      @size = 0
+
+      return results
     end
 
     def dispose
-      shutdown
-      join
+      @lock.synchronize do
+        shutdown
+        join
+      end
 
       return nil
     end
 
     def inspect
-      return "#<#{self.class.to_s}:0x#{(object_id << 1).to_s(16)} size=#{@size}>"
+      return "#<#{self.class.to_s}:0x#{(object_id << 1).to_s(16)} size=#{size}>"
+    end
+
+    def size
+      @lock.synchronize do
+        return @size
+      end
+    end
+
+    def expand(count)
+      @lock.synchronize do
+        count.times do
+            @workers << @worker_class.new(:input_queue => @input_queue)
+            @size += 1
+        end
+      end
+
+      return nil
+    end
+
+    def contract(count, &block)
+      @lock.synchronize do
+        raise 'Count is too large.' if count > @size
+
+        count.times do
+          callback = Proc.new do |worker|
+            remove_worker(worker)
+            block.call if block
+          end
+
+          enqueue(:shutdown, callback)
+          @size -= 1
+        end
+      end
+
+      return nil
+    end
+
+    def resize(new_size)
+      @lock.synchronize do
+        if new_size > @size
+          expand(new_size - @size)
+        elsif new_size < @size
+          contract(@size - new_size)
+        end
+      end
+
+      return nil
+    end
+
+    private
+
+    def remove_worker(worker)
+      @lock.synchronize do
+        @workers.delete(worker)
+      end
+
+      return nil
     end
   end
 end

data/lib/workers/scheduler.rb

@@ -8,6 +8,8 @@ module Workers
       @schedule = SortedSet.new
       @mutex = Mutex.new
       @thread = Thread.new { start_loop }
+
+      return nil
     end
 
     def schedule(timer)

data/lib/workers/timer.rb

@@ -11,12 +11,12 @@ module Workers
       @callback = options[:callback] || block
       @repeat = options[:repeat] || false
       @scheduler = options[:scheduler] || Workers.scheduler
-
       @mutex = Mutex.new
 
       reset
-
       @scheduler.schedule(self)
+
+      return nil
     end
 
     def <=>(other)

data/lib/workers/version.rb

@@ -1,3 +1,3 @@
 module Workers
-  VERSION = '0.0.9'
+  VERSION = '0.1.0'
 end

data/lib/workers/worker.rb

@@ -5,8 +5,9 @@ module Workers
     def initialize(options = {})
       @logger = Workers::LogProxy.new(options[:logger])
       @input_queue = options[:input_queue] || Queue.new
-
       @thread = Thread.new { start_event_loop }
+
+      return nil
     end
 
     def enqueue(command, data = nil)
@@ -64,25 +65,17 @@ module Workers
     end
 
     def shutdown_handler(event)
-      try_callback(event.data)
+      event.data.call(self) if event.data
     end
 
     def perform_handler(event)
-      try_callback(event.data)
+      event.data.call if event.data
     end
 
     def exception_handler(e)
       puts concat_e('Worker event loop died.', e)
     end
 
-    def try_callback(callback, &block)
-      begin
-        callback.call
-      rescue Exception => e
-        block.call(e) if block
-      end
-    end
-
     def process_event(event)
       raise "Unhandled event (#{event.inspect}). Subclass and override if you need custom events."
     end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: workers
 version: !ruby/object:Gem::Version
-  version: 0.0.9
+  version: 0.1.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-05-17 00:00:00.000000000 Z
+date: 2013-05-18 00:00:00.000000000 Z
 dependencies: []
 description: Simple Ruby workers for performing work in background threads.
 email: