task_tempest 0.1.0

data/.document

@@ -0,0 +1,5 @@
+README.rdoc
+lib/**/*.rb
+bin/*
+features/**/*.feature
+LICENSE

data/.gitignore

@@ -0,0 +1,21 @@
+## MAC OS
+.DS_Store
+
+## TEXTMATE
+*.tmproj
+tmtags
+
+## EMACS
+*~
+\#*
+.\#*
+
+## VIM
+*.swp
+
+## PROJECT::GENERAL
+coverage
+rdoc
+pkg
+
+## PROJECT::SPECIFIC

data/CHANGELOG

@@ -0,0 +1,2 @@
+0.1.0
+ - Initial version

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 Christopher J. Bottaro
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,132 @@
+= task_tempest
+
+Framework for building threaded asynchronous job processors.
+
+== Description
+
+task_tempest lets you build glorified loops.  You set some configuration options, define some callbacks, then it just loops, reading messages from a queue and processing them.
+
+You define the queue, how to read from the queue, how to write to the queue and the classes that process the queue messages.  The class that does the looping is called the tempest and the classes that handle job are called tasks.
+
+== Defining a tempest
+
+Defining a tempest is simple.  You just derive off of <tt>TaskTempest::Engine</tt>, set some configuration options and callbacks, then just instantiate the class and call +run+.
+
+  require "task_tempest"
+  class MyTempest < TaskTempest::Engine
+    process_name "my_tempest"
+  
+    queue do |logger|
+      logger.debug "initializing queue"
+      SuperCoolQueue.new(...)
+    end
+  
+    dequeue do |queue, logger|
+      message = queue.pop
+      if message
+        logger.info "message received"
+        YAML.load(message.body)
+      else
+        nil
+      end
+    end
+  end
+
++queue+ is given a logger object and required to return an instance of your queue.
+
++dequeue+ is given the queue object and a logger object and required to return a tuple <tt>[task_id, task_class_name, *task_args]</tt>.
+
+There are many more configuration options to set and callbacks to define, please see the rdocs.
+
+TODO provide rdocs.
+
+== Running a tempest
+
+You simply instantiate the tempest and call +run+.
+
+  MyTempest.new.run
+
+Catching an +Interrupt+ or +SystemExit+ exception will attempt a graceful shutdown, as will catching a <tt>SIGUSR2</tt> signal.  Catching a +SIGTERM+ signal will try to exit immediately.
+
+== Running as a daemon
+
+There is no code in +task_tempest+ to run as a daemon, that is left to you.  It's easy very easy with the {Daemons}[http://rubygems.org/gems/daemons] gem though.
+
+Assuming your tempest is defined in <tt>my_tempest.rb</tt>, just put the following code at the bottom of the file.
+
+  if $0 == __FILE__
+    require "daemons"
+    Daemons.run_proc(MyTempest.settings.process_name, :log_output => true) do
+      MyTempest.new.run
+    end
+  end
+
+Now you can run it as a daemon from the command line.
+
+  ruby my_tempest.rb start
+  ruby my_tempest.rb stop
+  ruby my_tempest.rb run # Run in foreground
+
+See the {rdoc}[http://daemons.rubyforge.org/] for {Daemons}[http://rubygems.org/gems/daemons] for more info.
+
+== Defining a task
+
+A task is what handles messages pulled off the queue.
+
+  require "task_tempest"
+  class GreeterTask < TaskTempest::Task
+    def start(person, greeting)
+      logger.info "about to greet #{person}"
+      puts "#{greeting}, #{person}!"
+    end
+  end
+
++start+ can take whatever arguments you want, but it must correspond with the arguments you put in the message.
+
+== Messages
+
+A message is what is returned by the +dequeue+ callback.  They are simply arrays of the form <tt>[task_id, task_class_name, *task_args]</tt>.  Note that if +task_id+ is nil, then one will be generated for automatically.
+
+An message like...
+
+  [nil, "GreeterTask", "Christopher", "Hello"]
+
+...would cause our +GreeterTask+ to puts "Hello, Christopher!".
+
+== Submitting tasks
+
+You can push messages on to the queue however you like, but +task_tempest+ provides a little convenience.  Assuming our previous examples...
+
+  task = GreeterTask.new("Christopher", "Hello")
+  MyTempest.submit(task)
+
+or
+
+  message = [nil, "GreeterTask", "Christopher", "Hello"]
+  MyTempest.submit(message)
+
+For this to work, you need to define the +enqueue+ callback in your tempest definition.
+
+  require "task_tempest"
+  class MyTempest < TaskTempest::Engine
+    ...
+    enqueue do |queue, message, logger, *args|
+      logger.info "enqueuing message #{message.inspect}"
+      queue.push(YAML.dump(message))
+    end
+    ...
+  end
+
+The +args+ argument is passed through via <tt>TaskTempest::Engine.submit</tt>.  For example, if you called +submit+ like...
+
+  MyTempest.submit(message, "one", "two")
+  
+Then +args+ would be <tt>["one", "two"]</tt>.  This is useful if you are using a priority queue and you want to submit your task with a given priority.
+
+== Complete example
+
+See the +examples+ directory.
+
+== Copyright
+
+Copyright (c) 2010 Christopher J. Bottaro. See LICENSE for details.

data/Rakefile

@@ -0,0 +1,53 @@
+require 'rubygems'
+require 'rake'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gem|
+    gem.name = "task_tempest"
+    gem.summary = %Q{Framework for creating asychronous job processors.}
+    gem.description = %Q{Framework for creating queue based, threaded asychronous job processors.}
+    gem.email = "cjbottaro@alumni.cs.utexas.edu"
+    gem.homepage = "http://github.com/cjbottaro/task_tempest"
+    gem.authors = ["Christopher J. Bottaro"]
+    # gem.add_development_dependency "thoughtbot-shoulda", ">= 0"
+    # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
+  end
+  Jeweler::GemcutterTasks.new
+rescue LoadError
+  puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
+end
+
+require 'rake/testtask'
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/test_*.rb'
+  test.verbose = true
+end
+
+begin
+  require 'rcov/rcovtask'
+  Rcov::RcovTask.new do |test|
+    test.libs << 'test'
+    test.pattern = 'test/**/test_*.rb'
+    test.verbose = true
+  end
+rescue LoadError
+  task :rcov do
+    abort "RCov is not available. In order to run rcov, you must: sudo gem install spicycode-rcov"
+  end
+end
+
+task :test => :check_dependencies
+
+task :default => :test
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+  version = File.exist?('VERSION') ? File.read('VERSION') : ""
+
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "task_tempest #{version}"
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end

data/VERSION

@@ -0,0 +1 @@
+0.1.0

data/examples/my_tempest.rb

@@ -0,0 +1,134 @@
+$LOAD_PATH << "../lib"
+
+require "rubygems"
+require "task_tempest"
+require "memcache"
+require "system_timer"
+
+require "tasks/evaler"
+require "tasks/greeter"
+
+class MemcachedQueue
+  
+  def initialize(name)
+    @name = name
+    @cache = MemCache.new "localhost:11211"
+  end
+  
+  def push(item)
+    queue = @cache.fetch(@name){ [] }
+    queue.push(item)
+    @cache.set(@name, queue)
+  end
+  
+  def pop
+    queue = @cache.fetch(@name){ [] }
+    item = queue.pop
+    @cache.set(@name, queue)
+    item
+  end
+  
+end
+
+# To run this example, open two shells and navagate to the examples directory (i.e. the
+# the directory containing this file).  In the first shell type:
+#   ruby my_tempest.rb run
+# In the second shell, invoke irb and type the following commands:
+#   require "my_tempest"
+#   MyTempest.submit(Greeter.new("Christopher", "Hello"))
+#   MyTempest.submit([nil, "Greeter", "Justin", "What up"])
+# Check the the first shell (and the logs dir) for output.
+# Note this example requires the SystemTimer, daemons and memcache-client gems.
+class MyTempest < TaskTempest::Engine
+  
+  # This dictates what the logs will be named.
+  process_name "my_tempest"
+  
+  # How many threads.
+  threads 5
+  
+  # Where to write the log files.
+  log_dir "log"
+  
+  # Where to look for task classes.  Will require each .rb file in this directory.
+  task_dir "tasks"
+  
+  # Time in seconds between each bookkeeping event.
+  bookkeeping_interval 15
+  
+  # Don't display log messages below this level.
+  log_level Logger::INFO
+  
+  # Maximum time in seconds a task is allowed to take before it is aborted.
+  task_timeout 5
+  
+  # What timeout method to use.  Timeout.timeout is unreliable.
+  timeout_method SystemTimer.method(:timeout_after)
+  
+  # Define the queue.
+  queue do |logger|
+    MemcachedQueue.new("my_tempest_queue")
+  end
+  
+  # Define how to enqueue messages.  This is used by MyTempest.submit.
+  # message is a tuple [task_id, task_class_name, *task_arguments].
+  # *args are passed through from MyTempest.submit.
+  enqueue do |queue, message, logger, *args|
+    logger.debug "enqueue #{message.inspect}"
+    queue.push(message)
+  end
+  
+  # Define how to dequeue messages.  It must return either
+  # nil or a tuple: [task_id, task_class_name, *task_arguments]
+  dequeue do |queue, logger|
+    if (message = queue.pop)
+      logger.debug "dequeue #{message.inspect}"
+      message
+    else
+      nil
+    end
+  end
+  
+  # Callback that happens after #init_logging, but before #bootstrap.
+  before_initialize do |logger|
+  end
+  
+  # Callback that happens after #bootstrap.
+  after_initialize do |logger|
+  end
+  
+  # Callback for an exception that happens in TaskTempest::Engine.
+  on_internal_exception do |e, logger|
+    puts "(I) #{e.class}: #{e.message}"
+  end
+  
+  # Callback that happens when an exception occurs in a task.
+  on_task_exception do |task, e, logger|
+    puts "(T:#{task_id}) #{e.class}: #{e.message}"
+  end
+  
+  # Callback that happens when a task exceeds the task_timeout setting.
+  on_task_timeout do |task, logger|
+    puts "(T:#{task.id}) timed out"
+  end
+  
+  # Callback that happens when a task calls Kernel.require.
+  on_require do |task, files, logger|
+    puts ("(T:#{task.id}) required files")
+  end
+  
+  # Callback that happens when bookkeeping is done.
+  on_bookkeeping do |book, logger|
+    if book[:files][:total_count] > 100
+      puts "you have a lot of open files!"
+    end
+  end
+  
+end
+
+if $0 == __FILE__
+  require "daemons"
+  Daemons.run_proc(MyTempest.settings.process_name, :log_output => true) do
+    MyTempest.new.run
+  end
+end

data/examples/tasks/evaler.rb

@@ -0,0 +1,7 @@
+class Evaler < TaskTempest::Task
+  
+  def start(code)
+    eval(code)
+  end
+  
+end

data/examples/tasks/greeter.rb

@@ -0,0 +1,8 @@
+class Greeter < TaskTempest::Task
+  
+  def start(person, greeting)
+    logger.info "#{person} has been greeted"
+    puts "#{greeting}, #{person}!"
+  end
+  
+end

data/lib/task_tempest/active_support.rb

@@ -0,0 +1,24 @@
+class Array
+  def separate(&block)
+    passed, failed = [], []
+    each do |item|
+      if block.call(item)
+        passed << item
+      else
+        failed << item
+      end
+    end
+    [passed, failed]
+  end unless method_defined?(:separate)
+end
+
+class Object
+  def metaclass
+    class << self; self; end
+  end unless method_defined?(:metaclass)
+  
+  def tap
+    yield self
+    self
+  end unless method_defined?(:tap)
+end

data/lib/task_tempest/bookkeeper.rb

@@ -0,0 +1,87 @@
+module TaskTempest
+  class Bookkeeper
+    attr_reader :storm
+    
+    def initialize(storm)
+      @storm = storm
+    end
+    
+    def book
+      return @book if @book
+      
+      book = {}
+      
+      executions = storm.clear_executions(:finished?)
+      ObjectSpace.garbage_collect
+      
+      # Task success/error counts.
+      book[:tasks] = {}
+      book[:tasks][:total_count] = executions.length
+      book[:tasks][:error_count] = executions.inject(0){ |memo, e| memo += 1 if e.exception; memo }
+      book[:tasks][:error_percentage] = begin
+        if book[:tasks][:total_count] > 0
+          book[:tasks][:error_count].to_f / book[:tasks][:total_count] * 100.0
+        else
+          0.0
+        end
+      end
+      book[:tasks][:per_thread] = tasks_per_thread(storm.threads, executions).values
+      book[:tasks][:avg_duration] = executions.inject(0){ |memo, e| memo += e.duration; memo }.to_f / executions.length
+      
+      # Thread (worker) info.
+      book[:threads] = {}
+      book[:threads][:busy] = storm.busy_workers.length
+      book[:threads][:idle] = storm.size - book[:threads][:busy]
+      book[:threads][:saturation] = book[:threads][:busy] / storm.size.to_f * 100 
+      
+      # Memory, Object, GC info.
+      book[:memory] = {}
+      book[:memory][:live_objects] = ObjectSpace.live_objects rescue nil
+      book[:memory][:resident] = get_memory(:resident)
+      book[:memory][:virtual] = get_memory(:virtual)
+      
+      # Open file counts.
+      book[:files] = {}
+      book[:files][:total_count] = get_files(:total)
+      book[:files][:tcp_count] = get_files(:tcp)
+      
+      @book = book
+    end
+    
+    # executions passed in are *finished*.
+    def tasks_per_thread(threads, executions)
+      counts_by_thread = threads.inject({}) do |memo, thread|
+        memo[thread] = 0
+        memo
+      end
+      executions.each do |e|
+        counts_by_thread[e.thread] += 1
+      end
+      counts_by_thread
+    end
+    
+    def get_memory(which)
+      @memory ||= `ps -o rss= -o vsz= -p #{Process.pid}`.split.collect{ |s| s.strip } rescue [nil, nil]
+      case which
+      when :resident
+        @memory[0].to_i
+      when :virtual
+        @memory[1].to_i
+      end
+    end
+    
+    def get_files(which)
+      @files ||= begin
+        output = `lsof -p #{Process.pid}` rescue ""
+        output.split("\n")
+      end
+      case which
+      when :total
+        @files.length
+      when :tcp
+        @files.inject(0){ |memo, line| memo += 1 if line.downcase =~ /tcp/; memo }
+      end
+    end
+    
+  end
+end

data/lib/task_tempest/bootstrap.rb

@@ -0,0 +1,77 @@
+module TaskTempest
+  module Bootstrap
+    
+    def self.included(mod)
+      mod.send(:extend, ClassMethods)
+      mod.send(:include, InstanceMethods)
+    end
+    
+    module ClassMethods
+    end
+    
+    module InstanceMethods
+      
+    private
+      
+      def bootstrap
+        init_logging
+        with_error_handling(:halt_on_error) do
+          init_thread_pool
+          before_initialize
+          init_tasks
+          init_queue
+          after_initialize
+          init_require
+        end
+      end
+      
+      def init_logging
+        @logger = Logger.new("#{settings.log_dir}/#{settings.process_name}.log")
+        @logger.formatter = LogFormatter
+        @logger.level = settings.log_level
+        logger.info "starting up"
+
+        @task_logger = Logger.new("#{settings.log_dir}/#{settings.process_name}.task.log")
+        @task_logger.formatter = LogFormatter
+        @task_logger.level = settings.log_level
+      end
+      
+      def init_tasks
+        logger.info "initializing tasks"
+        Dir.glob("#{settings.task_dir}/*.rb").each do |file_path|
+          logger.info file_path
+          require file_path
+        end
+      end
+
+      def init_queue
+        logger.info "initializing queue"
+        @queue = settings.queue.call(logger)
+      end
+
+      def init_thread_pool
+        logger.info "initializing thread pool"
+        @storm = ThreadStorm.new :size => settings.threads,
+                                 :reraise => false,
+                                 :timeout_method => settings.timeout_method,
+                                 :timeout => settings.task_timeout
+      end
+      
+      def before_initialize
+        logger.info "calling before_initialize"
+        settings.before_initialize.call(logger)
+      end
+      
+      def after_initialize
+        logger.info "calling after_initialize"
+        settings.after_initialize.call(logger)
+      end
+      
+      def init_require
+        require "task_tempest/require"
+      end
+      
+    end
+      
+  end
+end

data/lib/task_tempest/callbacks.rb

@@ -0,0 +1,29 @@
+module TaskTempest
+  module Callbacks
+    
+    def on_bookkeeping(book)
+      settings.on_bookkeeping.call(book, logger) if settings.on_bookkeeping
+    end
+    
+    def on_require(task, files)
+      return if files.empty?
+      logger.warn task.format_log "Kernel.require called on #{files.inspect}"
+      settings.on_require.call(task, files, logger)
+    end
+    
+    def on_internal_exception(e)
+      settings.on_internal_exception.call(e, logger)
+    rescue Exception => e
+      logger.error format_exception(e) rescue nil
+    end
+    
+    def on_task_exception(task, e)
+      settings.on_task_exception.call(task, e, logger)
+    end
+    
+    def on_task_timeout(task)
+      settings.on_task_timeout.call(task, logger)
+    end
+    
+  end
+end

data/lib/task_tempest/engine.rb

@@ -0,0 +1,168 @@
+require "thread_storm"
+
+require "task_tempest/bookkeeper"
+require "task_tempest/bootstrap"
+require "task_tempest/callbacks"
+require "task_tempest/error_handling"
+require "task_tempest/settings"
+
+module TaskTempest
+  class Engine
+    attr_reader :logger, :task_logger, :queue, :storm, :message, :tasks
+    
+    include Bootstrap
+    include Callbacks
+    include ErrorHandling
+    include Settings
+    
+    def self.inherited(derived)
+      derived.settings = settings.dup
+    end
+    
+    def self.submit_message(message, *args)
+      logger = Logger.new(STDOUT)
+      queue  = settings.queue.call(logger)
+      settings.enqueue.call(queue, message, logger, *args)
+    end
+    
+    def self.submit_task(task, *args)
+      submit_message(task.to_message, *args)
+    end
+    
+    def self.submit(task_or_message, *args)
+      if task_or_message.kind_of?(TaskTempest::Task)
+        submit_task(task_or_message, *args)
+      else
+        submit_message(task_or_message, *args)
+      end
+    end
+    
+    def initialize
+      @tasks = []
+      @bookkeeping_timer = Time.now
+    end
+    
+    def run
+      bootstrap
+      logger.info "starting run loop"
+      with_shutdown_handling{ heartbeat while true }
+    end
+    
+  private
+    
+    def heartbeat
+      with_error_handling{ receive_message }
+      with_error_handling{ dispatch_message }
+      with_error_handling{ finish_tasks }
+      with_error_handling{ bookkeeping }
+    end
+    
+    def receive_message
+      logger.debug "receiving message"
+      
+      if message
+        logger.debug "already have message"
+        return
+      end
+      
+      # Why do we do it this way?  Because of badly behaved dequeue
+      # definitions.  For example, right_aws rescues any exception
+      # when making a request to Amazon.  Thus if we try to shutdown
+      # our tempest, right_aws could potentially swallow that exception.
+      
+      @receive_storm ||= ThreadStorm.new :size => 1,
+                                         :timeout_method => settings.timeout_method,
+                                         :timeout => settings.dequeue_timeout
+      
+      execution = @receive_storm.execute{ settings.dequeue.call(queue, logger) }
+      with_error_handling do
+        @message = execution.value
+        logger.warn "dequeue timed out" if execution.timed_out?
+      end
+      @receive_storm.clear_executions # Prevent memory leak.
+        
+      if message.nil?
+        logger.debug "no available messages, sleeping for #{settings.no_message_sleep}"
+        sleep(settings.no_message_sleep)
+      end
+    end
+    
+    def dispatch_message
+      if storm.busy_workers.length == storm.size
+        logger.debug "no available threads, sleeping for #{settings.no_thread_sleep}"
+        sleep(settings.no_thread_sleep)
+      elsif message
+        dispatch_task
+      end
+    end
+    
+    def dispatch_task
+      id, name, *args = message
+      task = TaskTempest::Task.const_get(name).new(*args)
+      task.override :id => id, :logger => task_logger
+      task.spawn(storm)
+      tasks << task
+      logger.info task.format_log("started", true)
+      task.logger.info "arguments #{args.inspect}"
+    rescue Exception => e
+      raise
+    ensure
+      @message = nil # Ensure we pop a new message off the queue on next loop iteration.
+    end
+    
+    def finish_tasks
+      finished, @tasks = tasks.separate{ |task| task.execution.finished? }
+      finished.each{ |task| handle_finished_task(task) }
+    end
+    
+    def handle_finished_task(task)
+      if (e = task.execution.exception)
+        logger.info task.format_log("failed", true)
+        task.logger.fatal format_exception(e)
+        on_task_exception(task, e)
+      elsif task.execution.timed_out?
+        logger.info task.format_log("timed out", true)
+        on_task_timeout(task)
+      else
+        logger.info task.format_log("finished", true)
+        on_require(task, task.execution.value)
+      end
+    end
+    
+    def bookkeeping
+      # Return unless it's time to do bookkeeping.
+      if Time.now - @bookkeeping_timer > settings.bookkeeping_interval
+        @bookkeeping_timer = Time.now # Reset the timer.
+      else
+        return
+      end
+      
+      keeper = Bookkeeper.new(storm)
+      logger.info "[BOOKKEEPING] " + keeper.book.inspect
+      on_bookkeeping(keeper.book)
+    end
+    
+    def clean_shutdown
+      logger.info "shutting down"
+      begin
+        timeout(settings.shutdown_timeout) do
+          storm.join
+          storm.shutdown
+        end
+      rescue Timeout::Error => e
+        logger.warn "shutdown timeout exceeded"
+      end
+      finish_tasks
+      exit(0)
+    end
+    
+    def dirty_shutdown
+      exit(-1)
+    end
+    
+    def timeout(timeout, &block)
+      settings.timeout_method.call(timeout, &block)
+    end
+    
+  end
+end

data/lib/task_tempest/error_handling.rb

@@ -0,0 +1,52 @@
+module TaskTempest
+  module ErrorHandling
+    
+    SHUTDOWN_EXCEPTIONS = [
+      Interrupt,
+      SystemExit,
+      SignalException
+    ]
+    
+    def with_error_handling(halt_on_error = false)
+      yield
+    rescue *SHUTDOWN_EXCEPTIONS => e
+      raise
+    rescue Exception => e
+      on_internal_exception(e)
+      if halt_on_error
+        logger.fatal format_exception(e)
+        exit(-1)
+      else
+        logger.error format_exception(e)
+      end
+    end
+    
+    def with_shutdown_handling
+      yield
+    rescue *SHUTDOWN_EXCEPTIONS => e
+      if e.class == SignalException
+        handle_shutdown_signal(e) or raise
+      else
+        clean_shutdown
+      end
+    end
+    
+    def handle_shutdown_signal(e)
+      case e.message
+      when "SIGTERM"
+        logger.info "SIGTERM detected"
+        dirty_shutdown
+      when "SIGUSR2"
+        logger.info "SIGUSR2 detected"
+        clean_shutdown
+      else
+        false
+      end
+    end
+    
+    def format_exception(e)
+      "#{e.class} #{e.message}\n" + e.backtrace.join("\n")
+    end
+    
+  end
+end

data/lib/task_tempest/require.rb

@@ -0,0 +1,27 @@
+require "set"
+
+module Kernel
+  alias_method :original_require, :require
+  
+  def require(file)
+    without_ext = file.sub /(\.rb$)|(\.bundle$)/, ""
+    files = %w[.rb .bundle].collect{ |ext| without_ext + ext }
+    already_required = !($".to_set & files.to_set).empty?
+    required_files = Thread.current[:required_files]
+    required_files << file if required_files and not already_required
+    original_require(file)
+  end
+  
+  def self.record_requires!
+    if Thread.current[:required_files] == nil
+      Thread.current[:required_files] = []
+      yield
+      required_files = Thread.current[:required_files]
+      Thread.current[:required_files] = nil
+      required_files
+    else # Reentrant case.
+      yield
+      Thread.current[:required_files]
+    end
+  end
+end

data/lib/task_tempest/settings.rb

@@ -0,0 +1,144 @@
+require "logger"
+require "timeout"
+
+module TaskTempest
+  module Settings
+    
+    DEFAULTS = {
+      :process_name => "task_tempest",
+      :log_level => Logger::DEBUG,
+      :threads => 10,
+      :no_thread_sleep => 1,
+      :no_message_sleep => 1,
+      :task_timeout => nil,
+      :shutdown_timeout => 5, # 5 seconds
+      :dequeue_timeout => 2, # 2 seconds
+      :timeout_method => Timeout.method(:timeout),
+      :root_dir => File.expand_path(Dir.pwd),
+      :log_dir => File.expand_path(Dir.pwd),
+      :task_dir => File.expand_path(Dir.pwd),
+      :queue => nil,
+      :enqueue => Proc.new{ |queue, message| raise "not implemented" },
+      :dequeue => Proc.new{ |queue, logger| logger.error("dequeue not defined"); sleep(1); nil },
+      :bookkeeping_interval => 10*60, # 10 minutes
+      :before_initialize => Proc.new{ |logger| },
+      :after_initialize => Proc.new{ |logger| },
+      :on_internal_exception => Proc.new{ |e, logger| },
+      :on_task_exception => Proc.new{ |e, logger| },
+      :on_require => Proc.new{ |files, logger| },
+      :on_bookkeeping => Proc.new{ |book, logger| },
+      :on_task_timeout => Proc.new{ |task, logger| }
+    }
+    
+    def self.included(mod)
+      mod.metaclass.class_eval{ attr_accessor :settings }
+      mod.settings = Struct.new(*DEFAULTS.keys).new(*DEFAULTS.values)
+      mod.send(:include, InstanceMethods)
+      mod.send(:extend, ClassMethods)
+    end
+    
+    module InstanceMethods
+      
+      def settings
+        self.class.settings
+      end
+      
+    end
+    
+    module ClassMethods
+      
+      def process_name(value)
+        settings.process_name = value
+      end
+      
+      def log_level(value)
+        settings.log_level = value
+      end
+      
+      def threads(value)
+        settings.threads = value
+      end
+      
+      def no_message_sleep(value)
+        settings.no_message_sleep = value
+      end
+      
+      def no_thread_sleep(value)
+        settings.no_thread_sleep = value
+      end
+      
+      def root_dir(path)
+        settings.root_dir = File.expand_path(path)
+      end
+      
+      def log_dir(value)
+        settings.log_dir = File.expand_path(value)
+      end
+      
+      def task_dir(value)
+        settings.task_dir = File.expand_path(value)
+      end
+      
+      def timeout_method(value)
+        settings.timeout_method = value
+      end
+      
+      def dequeue_timeout(seconds)
+        settings.dequeue_timeout = value.to_f
+      end
+      
+      def task_timeout(value)
+        settings.task_timeout = value.to_f
+      end
+      
+      def shutdown_timeout(value)
+        settings.shutdown_timeout = value.to_f
+      end
+      
+      def queue(&block)
+        settings.queue = block
+      end
+      
+      def enqueue(&block)
+        settings.enqueue = block
+      end
+      
+      def dequeue(&block)
+        settings.dequeue = block
+      end
+      
+      def bookkeeping_interval(value)
+        settings.bookkeeping_interval = value
+      end
+      
+      def before_initialize(&block)
+        settings.before_initialize = block
+      end
+      
+      def after_initialize(&block)
+        settings.after_initialize = block
+      end
+      
+      def on_internal_exception(&block)
+        settings.on_internal_exception = block
+      end
+      
+      def on_task_exception(&block)
+        settings.on_task_exception = block
+      end
+      
+      def on_task_timeout(&block)
+        settings.on_task_timeout = block
+      end
+      
+      def on_require(&block)
+        settings.on_require = block
+      end
+      
+      def on_bookkeeping(&block)
+        settings.on_bookkeeping = block
+      end
+      
+    end
+  end
+end

data/lib/task_tempest/task.rb

@@ -0,0 +1,54 @@
+require "digest"
+require "logger"
+
+require "task_tempest/task_logger"
+require "task_tempest/require"
+
+module TaskTempest
+  class Task
+    attr_reader :id, :args, :execution
+    
+    def initialize(*args)
+      @id = generate_id
+      @args = args
+    end
+    
+    def override(options = {})
+      @id = options[:id] if options[:id]
+      @logger = TaskLogger.new(options[:logger], self) if options[:logger]
+    end
+    
+    def spawn(storm)
+      @execution = storm.execute{ run }
+    end
+    
+    def run
+      Kernel.record_requires!{ start(*args) }
+    end
+    
+    def start(*args)
+      raise "not implemented"
+    end
+    
+    def logger
+      @logger ||= TaskLogger.new(Logger.new(STDOUT), self)
+    end
+    
+    def to_message
+      [id, self.class.name, *args]
+    end
+    
+    def format_log(message, duration = false)
+      s = "{#{id}} <#{self.class}> #{message}"
+      s += " #{execution.duration}" if duration and execution.finished?
+      s
+    end
+    
+  private
+    
+    def generate_id
+      Digest::SHA1.hexdigest(Time.now.to_s + rand.to_s)[0,5]
+    end
+    
+  end
+end

data/lib/task_tempest/task_logger.rb

@@ -0,0 +1,18 @@
+module TaskTempest
+  class TaskLogger
+    
+    def initialize(logger, task)
+      @logger = logger
+      @task = task
+    end
+    
+    %w[debug info warn error fatal].each do |level|
+      class_eval <<-STR
+        def #{level}(msg)
+          @logger.#{level} @task.format_log(msg)
+        end
+      STR
+    end
+    
+  end
+end

data/lib/task_tempest.rb

@@ -0,0 +1,13 @@
+require "task_tempest/active_support"
+require "task_tempest/engine"
+require "task_tempest/task"
+
+module TaskTempest
+  
+  LogFormatter = Proc.new do |severity, time, progname, message|
+    message = message.call if message.respond_to?(:call)
+    time = time.strftime("%Y/%m/%d %H:%M:%S")
+    sprintf("%s [%s] %s\n", time, severity, message)
+  end
+  
+end

data/task_tempest.gemspec

@@ -0,0 +1,68 @@
+# Generated by jeweler
+# DO NOT EDIT THIS FILE DIRECTLY
+# Instead, edit Jeweler::Tasks in Rakefile, and run the gemspec command
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+  s.name = %q{task_tempest}
+  s.version = "0.1.0"
+
+  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
+  s.authors = ["Christopher J. Bottaro"]
+  s.date = %q{2010-06-24}
+  s.description = %q{Framework for creating queue based, threaded asychronous job processors.}
+  s.email = %q{cjbottaro@alumni.cs.utexas.edu}
+  s.extra_rdoc_files = [
+    "LICENSE",
+     "README.rdoc"
+  ]
+  s.files = [
+    ".document",
+     ".gitignore",
+     "CHANGELOG",
+     "LICENSE",
+     "README.rdoc",
+     "Rakefile",
+     "VERSION",
+     "examples/my_tempest.rb",
+     "examples/tasks/evaler.rb",
+     "examples/tasks/greeter.rb",
+     "lib/task_tempest.rb",
+     "lib/task_tempest/active_support.rb",
+     "lib/task_tempest/bookkeeper.rb",
+     "lib/task_tempest/bootstrap.rb",
+     "lib/task_tempest/callbacks.rb",
+     "lib/task_tempest/engine.rb",
+     "lib/task_tempest/error_handling.rb",
+     "lib/task_tempest/require.rb",
+     "lib/task_tempest/settings.rb",
+     "lib/task_tempest/task.rb",
+     "lib/task_tempest/task_logger.rb",
+     "task_tempest.gemspec",
+     "test/helper.rb",
+     "test/test_task_tempest.rb"
+  ]
+  s.homepage = %q{http://github.com/cjbottaro/task_tempest}
+  s.rdoc_options = ["--charset=UTF-8"]
+  s.require_paths = ["lib"]
+  s.rubygems_version = %q{1.3.7}
+  s.summary = %q{Framework for creating asychronous job processors.}
+  s.test_files = [
+    "test/helper.rb",
+     "test/test_task_tempest.rb",
+     "examples/my_tempest.rb",
+     "examples/tasks/evaler.rb",
+     "examples/tasks/greeter.rb"
+  ]
+
+  if s.respond_to? :specification_version then
+    current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
+    s.specification_version = 3
+
+    if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
+    else
+    end
+  else
+  end
+end
+

data/test/helper.rb

@@ -0,0 +1,10 @@
+require 'rubygems'
+require 'test/unit'
+require 'shoulda'
+
+$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
+$LOAD_PATH.unshift(File.dirname(__FILE__))
+require 'task_tempest'
+
+class Test::Unit::TestCase
+end

data/test/test_task_tempest.rb

@@ -0,0 +1,5 @@
+require 'helper'
+
+class TestTaskTempest < Test::Unit::TestCase
+  
+end

metadata

@@ -0,0 +1,95 @@
+--- !ruby/object:Gem::Specification 
+name: task_tempest
+version: !ruby/object:Gem::Version 
+  hash: 27
+  prerelease: false
+  segments: 
+  - 0
+  - 1
+  - 0
+  version: 0.1.0
+platform: ruby
+authors: 
+- Christopher J. Bottaro
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2010-06-24 00:00:00 -05:00
+default_executable: 
+dependencies: []
+
+description: Framework for creating queue based, threaded asychronous job processors.
+email: cjbottaro@alumni.cs.utexas.edu
+executables: []
+
+extensions: []
+
+extra_rdoc_files: 
+- LICENSE
+- README.rdoc
+files: 
+- .document
+- .gitignore
+- CHANGELOG
+- LICENSE
+- README.rdoc
+- Rakefile
+- VERSION
+- examples/my_tempest.rb
+- examples/tasks/evaler.rb
+- examples/tasks/greeter.rb
+- lib/task_tempest.rb
+- lib/task_tempest/active_support.rb
+- lib/task_tempest/bookkeeper.rb
+- lib/task_tempest/bootstrap.rb
+- lib/task_tempest/callbacks.rb
+- lib/task_tempest/engine.rb
+- lib/task_tempest/error_handling.rb
+- lib/task_tempest/require.rb
+- lib/task_tempest/settings.rb
+- lib/task_tempest/task.rb
+- lib/task_tempest/task_logger.rb
+- task_tempest.gemspec
+- test/helper.rb
+- test/test_task_tempest.rb
+has_rdoc: true
+homepage: http://github.com/cjbottaro/task_tempest
+licenses: []
+
+post_install_message: 
+rdoc_options: 
+- --charset=UTF-8
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.3.7
+signing_key: 
+specification_version: 3
+summary: Framework for creating asychronous job processors.
+test_files: 
+- test/helper.rb
+- test/test_task_tempest.rb
+- examples/my_tempest.rb
+- examples/tasks/evaler.rb
+- examples/tasks/greeter.rb