deferred 0.5.3

data/.gitignore

@@ -0,0 +1,5 @@
+*.gem
+.bundle
+Gemfile.lock
+pkg/*
+.idea

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--format documentation

data/Gemfile

@@ -0,0 +1,4 @@
+source "http://rubygems.org"
+
+# Specify your gem's dependencies in deferred.gemspec
+gemspec

data/MIT_LICENSE

@@ -0,0 +1,19 @@
+Copyright (C) 2011 by Hewlett Packard Development Company, L.P.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.markdown

@@ -0,0 +1,215 @@
+## Deferred: A bit of sugar to make async more pleasant
+
+Hey, EventMachine is pretty cool, scales really well, etc. The only thing is that you have to write all your code using deferreds, which can be kind of...challenging. This library will not solve all your problems, it will not walk your dog, it does not do windows. However, it provides syntactic sugar around some core EventMachine libraries to make things a bit more pleasant.
+
+### Deferred::InstanceMethods
+
+This module includes EM::Deferrable, and tweaks the behavior a little. First off the `callback` and `errback` methods will return `self` which allows for chaining. In addition, an `ensure`-like method has been added `ensure_that` that adds a block to both the `callback` and `errback` chains.
+
+```ruby
+client.do_something_eventually.callback do |arg|
+  something_happened(arg)
+end.errback do |err|
+  raise err
+end.ensure_that |*|
+  cleanup!
+end
+```
+
+Sometimes you have a deferred that needs to wait on another deferred to fire before it takes action, for this a `chain\_to` method is provided. 
+
+```ruby
+def something
+  Deferred::Default.new.tap do |my_dfr|
+    my_dfr.chain_to(client.do_something_eventually)
+  end
+end
+```
+
+EM::Deferred objects have a timeout method, but they don't let you specify an error class to use. In our code, all of our errbacks take an argument that is an Exception instance. Deferred::InstanceMethods#timeout allows you to optionally specify an Exception subclass that will be passed to registered errbacks after the given timeout is fired.
+
+```ruby
+  dfr = client.do_something_eventually
+  dfr.callback { |arg| something_happened!(arg) }
+  dfr.errback { |e| puts e.inspect }
+  dfr.timeout(3.0, MyTimeoutError)
+```
+
+In this case, if a timeout occurs, the errback would be called with a MyTimeoutError instance with the message "timeout after 3.0 seconds".
+
+Finally, there's the `errback\_on\_exception` method, that lets you fire the errback chain if an exception is raised in the given code block.
+
+```ruby
+dfr.errback_on_exception do
+  # try a bunch of stuff
+  raise "Oh noes! something bad!"
+end
+```
+
+In this case, registered errbacks would be called with a RuntimeError.
+
+
+### Deferred::Accessors
+
+A common pattern is providing lifecycle events on a given instance, clients connect, disconnect, startup, shutdown, error, etc. The `Deferred::Accessor.deferred\_event` class method provides a convenient way of declaring these deferreds in a class.
+
+```ruby
+class SomethingManager
+  include Deferred::Accessors
+
+  deferred_event :start
+
+  def start
+    # do stuff required for starting the SomethingManager
+   
+    on_start.succeed
+  end
+end
+```
+
+It's often more convenient to combine the callback with the call to initiate the action, for example:
+
+```ruby
+class SomethingManager
+  include Deferred::Accessors
+
+  deferred_event :start
+
+  def initialize
+  	@started = false
+  end
+
+  def start(&cb)
+    on_start(&cb) 	# registers cb as a callback
+
+   	return on_start if @started
+    @started = true
+
+	# this obviously would be some kind of deferred action that would
+	# eventually call on_start.succeed
+	#
+	EM.next_tick { on_start.succeed }
+   
+    on_start
+  end
+end
+```
+
+This allows one to do the following:
+
+```ruby
+s = SomethingManager.new
+s.start do 
+  do_something_when_manager_started
+end.errback do
+  raise "Oh Noes! Manager failed to start! Abort!"
+end
+```
+
+## Deferred::ThreadpoolJob
+
+A common need is to run some blocking task in the EventMachine threadpool using `EM.defer`. This works well, but there's no way of handling error cases, as when an exception is raised in the threadpool, the reactor dies. Enter the `ThreadpoolJob`, a Deferred that wraps the running of a job in the threadpool. 
+
+### The success case
+
+```ruby
+require 'rubygems'
+require 'eventmachine'
+require 'deferred'
+
+EM.run do
+  Deferred::DefaultThreadpoolJob.new.tap do |tpj|
+    tpj.before_run do
+      $stderr.puts "w00t! we're about to run in the threadpool, EM.reactor_thread? #{EM.reactor_thread?}"
+    end
+
+    tpj.on_run do
+      $stderr.puts "EM.reactor_thread? #{EM.reactor_thread?}"
+      $stderr.puts "sleeping for 0.5s"
+      sleep 0.5
+
+      %w[this is the result of the run]
+    end
+
+    tpj.callback do |*a|
+      $stderr.puts "Success, called with: #{a.inspect}"
+    end
+
+    tpj.errback do |exc|
+      $stderr.puts "Oh noes! an error happened: #{exc.inspect}"
+    end
+
+    # we have all the Deferred::InstanceMethods
+    tpj.ensure_that do |*|
+      EM.next_tick { EM.stop_event_loop }
+    end
+
+    tpj.defer!
+  end
+end
+```
+
+Produces the output:
+
+```
+w00t! we're about to run in the threadpool, EM.reactor_thread? true
+EM.reactor_thread? false
+sleeping for 0.5s
+Success, called with: [["this", "is", "the", "result", "of", "the", "run"]]
+```
+
+### The error case
+
+```ruby
+require 'rubygems'
+require 'eventmachine'
+require 'deferred'
+
+EM.run do
+  Deferred::DefaultThreadpoolJob.new.tap do |tpj|
+    tpj.before_run do
+      $stderr.puts "w00t! we're about to run in the threadpool, EM.reactor_thread? #{EM.reactor_thread?}"
+    end
+
+    tpj.on_run do
+      raise "Teh goggles, they do nothing!"
+    end
+
+    tpj.callback do |*a|
+      $stderr.puts "Success, called with: #{a.inspect}"
+    end
+
+    tpj.errback do |exc|
+      $stderr.puts "Oh noes! an error happened: #{exc.inspect}"
+    end
+
+    # we have all the Deferred::InstanceMethods
+    tpj.ensure_that do |*|
+      EM.next_tick { EM.stop_event_loop }
+    end
+
+    tpj.defer!
+  end
+end
+```
+
+Produces the output:
+
+```
+w00t! we're about to run in the threadpool, EM.reactor_thread? true
+Oh noes! an error happened: #<RuntimeError: Teh goggles, they do nothing!>
+```
+
+### Returning a deferred or an object that responds\_to?(:defer!)
+
+One use case we had was the ability for a ThreadpoolJob's on\_run block to return a deferred, which would be chained to the ThreadpoolJob's callbacks. This allows a lot of flexibility in terms of chaining tasks that should run both in and out of the threadpool. 
+
+This functionality was created for a task queueing system where *most* of the jobs had to run on the threadpool. Check out the rather [involved example][] which shows how to implement a DefaultThreadpoolJob that spawns several sub-jobs and waits for their completion before firing its own callbacks.
+
+
+## From the "Credit Where Credit is Due Dept." ##
+
+Thanks to [Snapfish][] for sponsoring development of this project and to HPDC, L.P. for agreeing to open source it.
+
+[Snapfish]: http://www.snapfish.com
+[involved example]: https://github.com/motionbox/deferred/blob/master/examples/threadpool_job_subtask.rb

data/Rakefile

@@ -0,0 +1 @@
+require 'bundler/gem_tasks'

data/deferred.gemspec

@@ -0,0 +1,23 @@
+# -*- encoding: utf-8 -*-
+$:.push File.expand_path("../lib", __FILE__)
+require "deferred/version"
+
+Gem::Specification.new do |s|
+  s.name        = "deferred"
+  s.version     = Deferred::VERSION
+  s.authors     = ["Jonathan D. Simms"]
+  s.email       = ["simms@hp.com"]
+  s.summary     = %q{Syntactical sugar around an EM::Deferrable}
+  s.description = s.summary + "\n"
+
+  s.add_dependency('eventmachine', '>= 0.12.10')
+
+  s.add_development_dependency('rspec', '>= 2.5.0', '< 3.0.0')
+  s.add_development_dependency('ZenTest', '>= 4.5.0')
+  s.add_development_dependency('evented-spec', '~> 0.4.1')
+
+  s.files         = `git ls-files`.split("\n")
+  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  s.require_paths = ["lib"]
+end

data/examples/threadpool_job_subtask.rb

@@ -0,0 +1,203 @@
+require 'rubygems'
+require 'eventmachine'
+require 'deferred'
+
+require 'logger'
+
+$log = Logger.new($stderr).tap { |l| l.level = Logger::DEBUG }
+
+# This is a real-world example of how we use this gem in a production setting.
+#
+# Surrounding this would be a daemon that's listening for events that represent
+# asynchronous tasks that need to be run (what the simplified TaskRunner class
+# below represents). We have components in our stack that are not evented, so we
+# need to run tasks on the threadpool. In addition, some of those tasks (like the
+# MainTask below) represent a "meta" task, a task that may have several
+# sub-components that can be run simultaneously. An example of this kind of task
+# would be one that needs to retrieve multiple resources from the web. The "meta"
+# task is "retrieve all sources," which can be divided into N number of tasks that
+# can be run concurrently.
+#
+# This code performs the following steps:
+#
+# * The TaskRunner creates a MainTask, and sets things up so that it can run its 
+#   process! method in the threadpool
+#
+# * The TaskRunner block is scheduled and the MainTask is run
+#
+# * The MainTask process! method:
+#   * sets up a callback that will call the handle_success method on successful
+#     completion of all subtasks (using the on_iterator_finished event)
+#   * sets up a callback that will call handle_failure in the case of any
+#     SubThreadTask failure (using the on_iterator_finished event)
+#   * sets up a next_tick block and returns self, effectively freeing the
+#     thread. (This is technically kind of a waste of the use of a thread, but
+#     for consistency with the rest of our task system, it's easier than
+#     special casing this)
+#
+# * Since MainTask is a Deferred, the TaskRunner will chain itself to the MainTask,
+#   meaning that when the MainTask calls back, the TaskRunner will inspect its return
+#   value and take the appropriate action (it will either be done, or if it's another
+#   Deferred it will wait on *that*, and if it's also a ThreadpoolJob it will
+#   call defer! on that return value). 
+#
+# * the reactor ticks
+#
+# * The next_tick block runs an EM::Iterator to create SubThreadTasks which
+#   will be run in their own threads. When all subtasks are complete it fires
+#   the on_iterator_finished event. If any SubThreadTask fails,
+#   on_iterator_finished will errback immediately, all running tasks will
+#   complete, but will be ignored.
+#
+# * The SubThreadTasks all complete
+#
+# * on_iterator_finished is fired and in order to handle the result MainTask will
+#   need to perform another ThreadpoolJob. We set up the ThreadpoolJob, and call
+#   succeed on the MainTask with that ThreadpoolJob as an argument. The MainTask
+#   will now depend on that job's succesful completion
+#
+# * the cleanup task runs in a separate thread
+#
+# * the cleanup task completes and the main TaskRunner job is done.
+#
+#
+# Now, this may seem like complete overkill, however it shows how flexible the
+# ThreadpoolJob is (credit to Topper Bowers who figured out the recursive
+# behavior for ThreadpoolJob#handle_result, ALL PRAISE TO THE HYPNOTOAD!). We
+# have a myriad of different solutions, some sync, some async. The flexibility
+# of Deferred allows us to leverage EventMachine to manage the overall
+# concurrency in a sensible way, but use the threadpool freely, and therefore
+# give us an incredibly powerful tool.
+#
+module ThreadpoolJobSubtasks
+  class SubThreadTask
+    include Deferred::ThreadpoolJob
+
+    attr_accessor :pause_for_sec
+
+    def initialize(pause_for_sec)
+      @pause_for_sec = pause_for_sec
+      @on_run_block = method(:process!).to_proc
+    end
+
+    def process!
+      $log.debug { "SubThreadTask: Thread id: %16x sleeping %0.9f sec" % [Thread.current.object_id, @pause_for_sec] }
+      sleep(@pause_for_sec)
+      self.succeed("SubThreadTask: Thread id: %16x complete %0.9f sec" % [Thread.current.object_id, @pause_for_sec] )
+    end
+  end
+
+  class MainTask
+    include Deferred
+    include Deferred::Accessors
+
+    deferred_event :iterator_finished
+
+    def initialize
+      @durations = []
+    end
+
+    def wait_for_durations=(durations)
+      @durations = durations
+    end
+
+    def process!
+      $log.debug { "MainTask#process! called, going to schedule sleeps for #{@durations.inspect} seconds concurrently" }
+
+      on_iterator_finished.callback { |*| handle_success }
+      on_iterator_finished.errback  { |e| handle_failure(e) }
+
+      EM.next_tick do
+        EM::Iterator.new(@durations.dup, 5).each(
+          lambda { |dur,iter|
+            tpj = SubThreadTask.new(dur)
+            tpj.before_run.callback { $log.debug { "SubThreadTask: about to sleep #{dur}" } }
+            tpj.callback { |s| $log.debug(s); @durations.delete(dur); iter.next }
+            tpj.errback  { |e| on_iterator_finished.fail(e) }
+            tpj.defer!
+          },
+          lambda { 
+            $log.debug { "MainTask: all SubThreadTasks completed, @durations: #{@durations.inspect}" }
+            on_iterator_finished.succeed 
+          }
+        )
+
+      end
+
+      self
+    end
+
+    # on success, we need to perform another blocking action, so we return a
+    # ThreadpoolJob instance (actually, anything that responds_to?(:defer!))
+    def handle_success
+      $log.debug { "MainTask: all SubThreadTasks have completed running successfully, now we do some other action on the threadpool before we're done" }
+
+      Deferred::DefaultThreadpoolJob.new.tap do |dtj|
+        dtj.on_run do
+          sleep(1.0)
+        end
+
+        dtj.before_run do
+          $log.debug { "MainTask: start some blocking action after success" }
+        end
+
+        dtj.callback do
+          $log.debug { "MainTask: some blocking action after success finished" }
+        end
+
+        dtj.errback { |e| raise e }
+
+        # here we do something fairly tricky.
+        # since this is all done for side-effect, we can signal that we're done processing
+        # (i.e. the main task is complete), but do some kind of asynchronous "next task"
+        # by returning a ThreadpoolJob.
+        $log.debug { "MainTask: succeeding the main task with another ThreadpoolJob" }
+
+        self.succeed(dtj)
+      end
+    end
+
+    def handle_failure(exc)
+      $log.debug { "Oh noes! an error occurred! #{exc.inspect}" }
+      self.fail(exc)
+    end
+  end
+
+  # This needs to run MainTask so that we don't get re-scheduled over and over
+  # (as if MainTask responds_to?(:defer!) it'll get deferred repeatedly)
+  class TaskRunner
+    include Deferred::ThreadpoolJob
+  end
+
+  def self.run!
+    sub_task_durations = []
+
+    20.times { sub_task_durations << rand }
+
+    EM.run do
+      main_task = MainTask.new.tap do |main|
+        main.wait_for_durations = sub_task_durations
+      end
+
+      TaskRunner.new.tap do |tr|
+        tr.on_run { main_task.process! }
+
+        tr.callback { $log.debug { "TaskRunner: main task completed" } }
+        tr.errback  { |e| raise e }
+
+        tr.ensure_that do 
+          EM.next_tick do
+            $log.debug { "TaskRunner: we're outta here" }
+            EM.stop_event_loop
+          end
+        end
+
+        tr.defer!
+      end
+    end
+  end
+end
+
+ThreadpoolJobSubtasks.run!
+
+