workers 0.1.4 → 0.2.0

data/README.md

@@ -19,11 +19,29 @@ Or install it yourself as:
 
     $ gem install workers
 
+## Parallel Map
+
+Parallel map is the simplest way to get started with the Workers gem.
+It is similar to Ruby's built in Array#map method except each element is mapped in parallel.
+
+    Workers.map([1, 2, 3, 4, 5]) { |i| i * i }
+
+Any exceptions while mapping with cause the entire map method to fail.
+If your block is prone to temporary failures (exceptions), you can retry it.
+
+    Workers.map([1, 2, 3, 4, 5], :max_tries => 100) do |i|
+      if rand <= 0.5
+        puts "Sometimes I like to fail while computing #{i} * #{i}."
+        raise 'sad face'
+      end
+
+      i * i
+    end
+
 ## Tasks
 
-Tasks and task groups build on top of pools of worker threads.
-They provide a means of parallelizing expensive computations, collecing results, and handling exceptions.
-These are the classes you normally work with in your application level code because they remove boiletplate.
+Tasks and task groups provide more flexibility than parallel map.
+The main benefit is that you get to decide how you want to handle exceptions.
 
     # Create a task group (it contains a pool of worker threads).
     group = Workers::TaskGroup.new
@@ -31,7 +49,7 @@ These are the classes you normally work with in your application level code beca
     # Add tasks to the group.
     10.times do |i|
       10.times do |j|
-        group.add(i, j) do
+        group.add do
           group.synchronize { puts "Computing #{i} * #{j}..." }
           i * j # Last statement executed is the result of the task.
         end
@@ -63,25 +81,26 @@ These are the classes you normally work with in your application level code beca
 Note that instances of TaskGroup provide a 'synchronize' method.
 This method uses a mutex so you can serialize portions of your tasks that aren't thread safe.
 
-## Parallel Map
-
-Parallel Map is syntactic sugar for tasks and task groups:
-
-    group = Workers::TaskGroup.new
-    group.map([1,2,3,4,5]) { |i| i * i }
+#### Options
 
-The above syntax is equivalent to the below:
+    group = Workers::TaskGroup.new(
+      :logger => nil,                   # Ruby logger instance.
+      :pool => Workers.pool             # The workers pool used to execute timer callbacks.
+    )
 
-    Workers.map([1, 2, 3, 4, 5]) { |i| i * i }
+    task = Workers::Task.new(
+      :logger => nil,                   # Ruby logger instance.
+      :perform => proc {},              # Required option.  Block of code to run.
+      :args => [],                      # Array of arguments passed to the 'perform' block.
+      :finished => nil,                 # Callback to execute after attempting to run the task.
+      :max_tries => 1,                  # Number of times to try completing the task (without an exception).
+    )
 
-Note that any task failures (exceptions) will cause an exception to be raised.
-This means that parallel map is "all or nothing" where as tasks and task groups leave error processing up to you.
+## Workers
 
-## Workers - Basic
+#### Basic
 
-There are many cases where tasks, task groups, and parallel map are too high-level.
-Fortunately you can use the lower level Worker class to solve these problems.
-The main purpose of the Worker class is to add an event system on top of the Thread class.
+The main purpose of the Worker class is to add an event system on top of Ruby's built-in Thread class.
 This greatly simplifies inter-thread communication.
 
     # Initialize a worker pool.
@@ -103,7 +122,7 @@ This greatly simplifies inter-thread communication.
     # Wait for the workers to shutdown.
     pool.join
 
-## Workers - Advanced
+#### Advanced
 
 The Worker class is designed to be customized through inheritence and its event system:
 
@@ -138,6 +157,13 @@ The Worker class is designed to be customized through inheritence and its event
 Note that you can use custom workers without a pool.
 This effectively gives you direct access to a single event-driven thread.
 
+#### Options
+
+    worker = Workers::Worker.new(
+      :logger => nil,                   # Ruby Logger instance.
+      :input_queue => nil               # Ruby Queue used for input events.
+    )
+
 ## Pools
 
 As shown above, pools effectively allow a group of workers to share a work queue.
@@ -158,6 +184,14 @@ They have a few additional methods described below:
     # Resize the pool size to a specific value.
     pool.resize(20)
 
+#### Options
+
+    pool = Workers::Pool.new(
+      :size => 20,                      # Number of threads to create.
+      :logger => nil,                   # Ruby Logger instance.
+      :worker_class => Workers::Worker  # Class of worker to use for this pool.
+    )
+
 ## Timers
 
 Timers provide a way to execute code in the future:
@@ -178,6 +212,21 @@ Timers provide a way to execute code in the future:
     # Shutdown the timer.
     timer.cancel
 
+#### Options
+
+    timer = Workers::Timer.new(1,
+      :logger => nil,                   # Ruby logger instance.
+      :repeat => false,                 # Repeat the timer until 'cancel' is called.
+      :scheduler => Workers.scheduler,  # The scheduler that manages execution.
+      :callback => nil                  # The proc to execute (provide this or a block, but not both).
+    )
+
+    timer = Workers::PeriodicTimer.new(1,
+      :logger => nil,                   # Ruby logger instance.
+      :scheduler => Workers.scheduler,  # The scheduler that manages execution.
+      :callback => nil                  # The proc to execute (provide this or a block, but not both).
+    )
+
 ## Schedulers
 
 Schedulers are what trigger Timers to fire.
@@ -201,43 +250,7 @@ You can create additional or custom ones as necessary:
     # Shutdown the scheduler.
     scheduler.dispose
 
-## Options (defaults below):
-
-    pool = Workers::Pool.new(
-      :size => 20,                      # Number of threads to create.
-      :logger => nil,                   # Ruby Logger instance.
-      :worker_class => Workers::Worker  # Class of worker to use for this pool.
-    )
-
-    worker = Workers::Worker.new(
-      :logger => nil,                   # Ruby Logger instance.
-      :input_queue => nil               # Ruby Queue used for input events.
-    )
-
-    timer = Workers::Timer.new(1,
-      :logger => nil,                   # Ruby logger instance.
-      :repeat => false,                 # Repeat the timer until 'cancel' is called.
-      :scheduler => Workers.scheduler,  # The scheduler that manages execution.
-      :callback => nil                  # The proc to execute (provide this or a block, but not both).
-    )
-
-    group = Workers::TaskGroup.new(
-      :logger => nil,                   # Ruby logger instance.
-      :pool => Workers.pool             # The workers pool used to execute timer callbacks.
-    )
-
-    task = Workers::Task.new(
-      :logger => nil,                   # Ruby logger instance.
-      :perform => proc {},              # Required option.  Block of code to run.
-      :args => [],                      # Array of arguments passed to the 'perform' block.
-      :finished => nil,                 # Callback to execute after attempting to run the task.
-    )
-
-    timer = Workers::PeriodicTimer.new(1,
-      :logger => nil,                   # Ruby logger instance.
-      :scheduler => Workers.scheduler,  # The scheduler that manages execution.
-      :callback => nil                  # The proc to execute (provide this or a block, but not both).
-    )
+#### Options
 
     scheduler = Workers::Scheduler.new(
       :logger => nil,                   # Ruby logger instance.

data/lib/workers/task.rb

@@ -12,7 +12,11 @@ module Workers
       @args = options[:args] || []
       @perform = options[:perform] || raise('Perform callback is required.')
       @finished = options[:finished]
+      @max_tries = options[:max_tries] || 1
       @state = :initialized
+      @tries = 0
+
+      raise 'max_tries must be >= 1' unless @max_tries >= 1
 
       return nil
     end
@@ -22,12 +26,17 @@ module Workers
 
       @state = :running
 
-      begin
-        @result = @perform.call(*@args)
-        @state = :succeeded
-      rescue Exception => e
-        @state = :failed
-        @exception = e
+      while(@tries < @max_tries && @state != :succeeded)
+        @tries += 1
+
+        begin
+          @result = @perform.call(*@args)
+          @state = :succeeded
+          @exception = nil
+        rescue Exception => e
+          @state = :failed
+          @exception = e
+        end
       end
 
       @finished.call(self)

data/lib/workers/task_group.rb

@@ -18,14 +18,13 @@ module Workers
       return nil
     end
 
-    def add(*args, &block)
+    def add(options = {}, &block)
       state!(:initialized)
 
-      if args[0].is_a?(Workers::Task)
-        @tasks << args[0]
-      else
-        @tasks << Workers::Task.new(:args => args, :perform => block, :finished => method(:finished))
-      end
+      options[:finished] = method(:finished)
+      options[:perform] ||= block
+
+      @tasks << Workers::Task.new(options)
 
       return nil
     end
@@ -57,9 +56,9 @@ module Workers
       return @tasks.select { |t| t.failed? }
     end
 
-    def map(inputs, &block)
+    def map(inputs, options = {}, &block)
       inputs.each do |input|
-        add(input) do |i|
+        add(:args => input, :max_tries => options[:max_tries]) do |i|
           block.call(i)
         end
       end

data/lib/workers/version.rb

@@ -1,3 +1,3 @@
 module Workers
-  VERSION = '0.1.4'
+  VERSION = '0.2.0'
 end

data/lib/workers.rb

@@ -33,9 +33,9 @@ module Workers
     @scheduler = val
   end
 
-  def self.map(vals, &block)
-    return Workers::TaskGroup.new.map(vals) do |v|
-      block.call(v)
+  def self.map(inputs, options = {}, &block)
+    return Workers::TaskGroup.new.map(inputs, options) do |i|
+      block.call(i)
     end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: workers
 version: !ruby/object:Gem::Version
-  version: 0.1.4
+  version: 0.2.0
   prerelease: 
 platform: ruby
 authors: