simple_job 0.0.3 → 0.1.0

data/CHANGELOG.rdoc

@@ -1,5 +1,11 @@
 = Simple Job
 
+== Version 0.1.0
+* Added AWS CloudWatch monitors for job execution
+* Added :replace_existing option to register_simple_job
+* Added signal handling to SQSJobQueue#poll to allow clean exit
+* New SQSJobQueue#poll implementation to allow metric gathering
+
 == Version 0.0.3
 * Properly handling SystemExit and SignalException in SQSJobQueue#poll so that process can be daemonized
 

data/README.rdoc

@@ -6,6 +6,8 @@ A gem containing libraries that support running background jobs or tasks. It's d
 * Enqueue a job
 * Poll for and execute jobs
 
+It fits seamlessly into a rails environment (utilizing its configuration), but does not require one.
+
 It is architected to support multiple types of queue implementations, but currently, it only includes an implementation using AWS SQS (http://aws.amazon.com/sqs/). Alternate queue implementations could be plugged in by the client using lib/simple_job/sqs_job_queue.rb as an example.
 
 The AWS SQS queue implementation requires the aws-sdk gem, which must be initialized (by calling AWS.config) for this API to be capable of enqueuing or polling for jobs.
@@ -24,17 +26,54 @@ SQSJobQueue is the queue implementation used by default. This may be overridden
  SimpleJob::SQSJobQueue.config :queue_prefix => 'my-job'
  SimpleJob::SQSJobQueue.define_queue 'normal', :default => true
 
-=== Complex configuration with explicit queue implementation, non-rails-defined environment, and multiple queues
+=== Complex configuration with explicit queue implementation, non-rails-defined environment, multiple queues, and CloudWatch monitoring
 
  SimpleJob::JobQueue.config :implementation => 'sqs'
- SimpleJob::SQSJobQueue.config :queue_prefix => 'my-job', :environment => 'production'
+ SimpleJob::SQSJobQueue.config :queue_prefix => 'my-job', :environment => 'production', :cloud_watch_namespace => 'SimpleJob'
  SimpleJob::SQSJobQueue.define_queue 'normal', :visibility_timeout => 60, :default => true
  SimpleJob::SQSJobQueue.define_queue 'high-priority', :visibility_timeout => 10
  SimpleJob::SQSJobQueue.define_queue 'long-running', :visibility_timeout => 3600
 
+=== AWS CloudWatch monitoring
+
+If you provide a CloudWatch namespace to SimpleJob::SQSJobQueue.config, then a series of metrics will be saved to AWS CloudWatch by the queue's poll method:
+
+ SimpleJob::SQSJobQueue.config :cloud_watch_namespace => 'MyNamespace'
+
+The following metrics will be published:
+
+MessageCheckCount:: A count of the number of poll attempts
+MessageReceivedCount:: A count of the number of poll attempts that return a message
+MessageMissCount:: A count of the number of poll attempts that do not return a message
+ExecutionCount:: A count of the number of jobs that are executed
+SuccessCount:: A count of the number of jobs that are executed without error
+ErrorCount:: A count of the number of jobs that raise an exception during execution
+ExecutionTime:: The number of milliseconds it takes the job to execute
+TimeToCompletion:: The number milliseconds between when a job is requested and when it is successfully completed
+ExecutionAttempts:: The number of executions that occur before a job is successfully completed
+
+The metrics above are published using the following dimensions:
+
+Environment:: The current environment (ie. development vs. production)
+SQSQueueName:: The name of the SQS queue being polled
+Host:: The hostname of the machine running the poll operation
+JobType:: The string label for the type of job being executed
+
+Note that the JobType dimension will only be present on the ExecutionCount, SuccessCount, ErrorCount, ExecutionTime, TimeToCompletion, and ExecutionAttempts metrics.
+
+If no namespace is provided to SimpleJob::SQSJobQueue.config, then no cloudwatch metrics will be gathered or published.
+
 
 == Job Definition
 
+To define a job, simply include the SimpleJob::JobDefinition module, and implement its #execute method.
+
+Attributes declared using simple_job_attribute will be automatically serialized into the JSON message that's enqueued, and re-populated into the object once it's dequeued. They each have standard getters/setters defined by attr_accessor, and may be set upon object initialization by passing them as hash keys/values to #new.
+
+ActiveModel validation is supported and included automatically.
+
+=== Synopsis
+
  class FooSender
    include SimpleJob::JobDefinition
 
@@ -53,7 +92,9 @@ SQSJobQueue is the queue implementation used by default. This may be overridden
 
 === Typical usage of default queue
 
-You may call #enqueue with no arguments, in which case JobQueue.default will be used.
+You may call #enqueue with no arguments, in which case JobQueue.default will be used (defined by passing the :default option when defining the queue as documented in the QueueConfiguration section).
+
+Similar to ActiveRecord's save method, enqueue will return true or false depending on whether the object passes validation.
 
  f = FooSender.new(:target => 'joe', :foo_content => 'foo!')  # can also assign attributes with f.target=, f.foo_content=
  if f.enqueue
@@ -87,11 +128,11 @@ Alternatively, the queue may be attached to the job upfront for multiple enqueue
 
 == Job Server Usage
 
-Calling #poll on a queue instance will, by default, dispatch each job to the proper registered class based on type and version. Its options are passed through to the underlying implementation. In the case of the SQSJobQueue, poll accepts the options documented for the aws-sdk AWS::SQS::Queue#poll method.
+Calling #poll on a queue instance will, by default, dispatch each job to the proper registered class based on type and version. Its options are passed through to the underlying implementation. See SQSJobQueue::poll for documentation of the options it accepts.
 
  JobQueue.default.poll(:poll_interval => 1, :idle_timeout => 60)
 
-You can override the default behavior by passing a block to #poll that accepts both a job definition instance and an SQS message.
+You can override the default behavior by passing a block to #poll that accepts both a job definition instance and an SQS message. In this case, you must call "definition.execute" in the block. The default handler simply calls definition.execute.
 
  JobQueue.default.poll(:poll_interval => 1, :idle_timeout => 60) do |definition, message|
    logger.debug "received message: #{message.inspect}"

data/lib/simple_job/job_definition.rb

@@ -127,7 +127,9 @@ module JobDefinition
     end
 
     def register_simple_job(options = {})
-      default_type = self.name.underscore.to_sym
+      default_type = self.name.split('::').last.underscore.to_sym
+
+      replace_existing = options.delete(:replace_existing) || true
 
       @definition = {
         :class => self,
@@ -139,7 +141,10 @@ module JobDefinition
       @definition[:versions] = Array(@definition[:versions])
       @definition[:versions].collect! { |value| value.to_s }
 
-      ::SimpleJob::JobDefinition.job_definitions.delete_if { |item| item[:type] == default_type }
+      if replace_existing
+        ::SimpleJob::JobDefinition.job_definitions.delete_if { |item| item[:type] == default_type }
+      end
+
       ::SimpleJob::JobDefinition.job_definitions << @definition
     end
 

data/lib/simple_job/sqs_job_queue.rb

@@ -1,6 +1,13 @@
+require 'socket'
+require 'fog'
+
 module SimpleJob
+
+# A SimpleJob::JobQueue implementation that uses AWS SQS
 class SQSJobQueue < JobQueue
 
+  DEFAULT_POLL_INTERVAL = 1
+
   # Registers this queue implementation with SimpleJob::JobQueue with identifier "sqs".
   register_job_queue 'sqs', self
 
@@ -9,6 +16,7 @@ class SQSJobQueue < JobQueue
       :queue_prefix => ENV['SIMPLE_JOB_SQS_JOB_QUEUE_PREFIX'],
       :default_visibility_timeout => 60,
       :environment => (defined?(Rails) && Rails.env) || 'development',
+      :cloud_watch_namespace => nil,
     }
 
     @config.merge!(options) if options
@@ -49,33 +57,88 @@ class SQSJobQueue < JobQueue
     sqs_queue.send_message(message)
   end
 
+  # Polls the queue, matching incoming messages with registered jobs, and
+  # executing the proper job/version.
+  #
+  # If called without a block, it will simply call the #execute method of the
+  # matched job. A block may be passed to add custom logic, but in this case
+  # the caller is responsible for calling #execute. The block will be passed
+  # two arguments, the matching job definition (already populated with the
+  # contents of the message) and the raw AWS message.
+  #
+  # The queue's configured visibility timeout will be used unless the
+  # :visibility_timeout option is passed (as a number of seconds).
+  #
+  # By default, the message's 'sent_at', 'receive_count', and
+  # 'first_received_at' attributes will be populated in the AWS message, but
+  # this may be overridden by passing an array of symbols to the :attributes
+  # option.
+  #
+  # By default, errors during job execution or message polling will be logged
+  # and the polling will continue, but that behavior may be changed by setting
+  # the :raise_exceptions option to true.
+  #
+  # By defult, this method will poll indefinitely. If you pass an :idle_timeout
+  # option, the polling will stop and this method will return if that number
+  # of seconds passes without receiving a message. In both cases, the method
+  # will safely complete processing the current message and return if a HUP,
+  # INT, or TERM signal is sent to the process.
+  #
+  # Note that this method will override any signal handlers for the HUP, INT,
+  # or TERM signals during its execution, but the previous handlers will be
+  # restored once the method returns.
   def poll(options = {}, &block)
     options = {
       :visibility_timeout => visibility_timeout,
       :attributes => [ :sent_at, :receive_count, :first_received_at ],
       :raise_exceptions => false,
+      :idle_timeout => nil,
+      :poll_interval => DEFAULT_POLL_INTERVAL
     }.merge(options)
 
     message_handler = block || lambda do |definition, message|
       definition.execute
     end
 
+    exit_next = false
+
+    logger.debug 'trapping terminate signals with function to exit loop'
+    signal_exit = lambda do
+      logger.info "caught signal to shutdown; finishing current message and quitting..."
+      exit_next = true
+    end
+    previous_traps = {}
+    ['HUP', 'INT', 'TERM'].each do |signal|
+      previous_traps[signal] = Signal.trap(signal, signal_exit)
+    end
+
+    last_message_at = Time.now
+
     loop do
       last_message = nil
+      current_start_milliseconds = get_milliseconds
+      current_job_type = 'unknown'
       begin
-        sqs_queue.poll(options) do |message|
+        sqs_queue.receive_messages(options) do |message|
           last_message = message
           raw_message = JSON.parse(message.body)
+          current_job_type = raw_message['type']
           definition_class = JobDefinition.job_definition_class_for(raw_message['type'], raw_message['version'])
           raise('no definition found') if !definition_class
           definition = definition_class.new.from_json(message.body)
           message_handler.call(definition, message)
         end
-        return
-      rescue SignalException, SystemExit => e
-        logger.info "received #{e.class}; exiting poll loop and re-raising: #{e.message}"
-        raise e
+
+        log_execution(true, last_message, current_job_type, current_start_milliseconds)
+
+        break if options[:idle_timeout] && ((Time.now - last_message_at) > options[:idle_timeout])
+
+        unless last_message
+          Kernel.sleep(options[:poll_interval]) unless options[:poll_interval] == 0
+        end
       rescue Exception => e
+        log_execution(false, last_message, current_job_type, current_start_milliseconds)
+
         if options[:raise_exceptions]
           raise e
         else
@@ -84,7 +147,15 @@ class SQSJobQueue < JobQueue
           logger.error(e.backtrace.join("\n  "))
         end
       end
+      break if exit_next
+    end
+
+    logger.debug 'restoring previous signal traps'
+    previous_traps.each do |signal, command|
+      Signal.trap(signal, command)
     end
+
+    logger.info "shutdown successful"
   end
 
   private
@@ -93,17 +164,125 @@ class SQSJobQueue < JobQueue
     attr_accessor :queues
   end
 
-  attr_accessor :sqs_queue, :visibility_timeout
+  attr_accessor :queue_name, :sqs_queue, :visibility_timeout, :cloud_watch
 
   def initialize(type, visibility_timeout)
     sqs = ::AWS::SQS.new
-    self.sqs_queue = sqs.queues.create "#{self.class.config[:queue_prefix]}-#{type}-#{self.class.config[:environment]}"
+    self.queue_name = "#{self.class.config[:queue_prefix]}-#{type}-#{self.class.config[:environment]}"
+    self.sqs_queue = sqs.queues.create(queue_name)
     self.visibility_timeout = visibility_timeout
+    self.cloud_watch = Fog::AWS::CloudWatch.new(
+      :aws_access_key_id => AWS.config.access_key_id,
+      :aws_secret_access_key => AWS.config.secret_access_key
+    )
   end
 
   def logger
     JobQueue.config[:logger]
   end
 
+  def get_milliseconds(time = Time.now)
+    (time.to_f * 1000).round
+  end
+
+  def log_execution(successful, message, job_type, start_milliseconds)
+    if self.class.config[:cloud_watch_namespace]
+      timestamp = DateTime.now.to_s
+      environment = self.class.config[:environment]
+      hostname = Socket.gethostbyname(Socket.gethostname).first
+
+      message_dimensions = [
+        { 'Name' => 'Environment', 'Value' => environment }, 
+        { 'Name' => 'SQSQueueName', 'Value' => queue_name },
+        { 'Name' => 'Host', 'Value' => hostname },
+      ]
+
+      job_dimensions = message_dimensions + [
+        { 'Name' => 'JobType', 'Value' => job_type },
+      ]
+
+      metric_data = [
+        {
+          'MetricName' => 'MessageCheckCount',
+          'Timestamp' => timestamp,
+          'Unit' => 'Count',
+          'Value' => 1,
+          'Dimensions' => message_dimensions
+        },
+        {
+          'MetricName' => 'MessageReceivedCount',
+          'Timestamp' => timestamp,
+          'Unit' => 'Count',
+          'Value' => message ? 1 : 0,
+          'Dimensions' => message_dimensions
+        },
+        {
+          'MetricName' => 'MessageMissCount',
+          'Timestamp' => timestamp,
+          'Unit' => 'Count',
+          'Value' => message ? 0 : 1,
+          'Dimensions' => message_dimensions
+        }
+      ]
+       
+      if message
+        now = get_milliseconds
+
+        metric_data.concat([
+          {
+            'MetricName' => 'ExecutionCount',
+            'Timestamp' => timestamp,
+            'Unit' => 'Count',
+            'Value' => 1,
+            'Dimensions' => job_dimensions
+          },
+          {
+            'MetricName' => 'SuccessCount',
+            'Timestamp' => timestamp,
+            'Unit' => 'Count',
+            'Value' => successful ? 1 : 0,
+            'Dimensions' => job_dimensions
+          },
+          {
+            'MetricName' => 'ErrorCount',
+            'Timestamp' => timestamp,
+            'Unit' => 'Count',
+            'Value' => successful ? 0 : 1,
+            'Dimensions' => job_dimensions
+          },
+          {
+            'MetricName' => 'ExecutionTime',
+            'Timestamp' => timestamp,
+            'Unit' => 'Milliseconds',
+            'Value' => now - start_milliseconds,
+            'Dimensions' => job_dimensions
+          }
+        ])
+
+        if successful
+          metric_data.concat([
+            {
+              'MetricName' => 'TimeToCompletion',
+              'Timestamp' => timestamp,
+              'Unit' => 'Milliseconds',
+              'Value' => now - get_milliseconds(message.sent_at),
+              'Dimensions' => job_dimensions
+            },
+            {
+              'MetricName' => 'ExecutionAttempts',
+              'Timestamp' => timestamp,
+              'Unit' => 'Count',
+              'Value' => message.receive_count,
+              'Dimensions' => job_dimensions
+            }
+          ])
+        end
+      end
+
+      cloud_watch.put_metric_data(self.class.config[:cloud_watch_namespace], metric_data)
+    end
+  end
+
 end
 end
+

data/lib/simple_job/version.rb

@@ -1,3 +1,3 @@
 module SimpleJob
-  VERSION = '0.0.3'
+  VERSION = '0.1.0'
 end

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: simple_job
 version: !ruby/object:Gem::Version 
-  hash: 25
+  hash: 27
   prerelease: 
   segments: 
   - 0
+  - 1
   - 0
-  - 3
-  version: 0.0.3
+  version: 0.1.0
 platform: ruby
 authors: 
 - David Dawson
@@ -62,6 +62,21 @@ dependencies:
   name: aws-sdk
   prerelease: false
   type: :runtime
+- !ruby/object:Gem::Dependency 
+  requirement: &id004 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        hash: 13
+        segments: 
+        - 1
+        - 1
+        version: "1.1"
+  version_requirements: *id004
+  name: fog
+  prerelease: false
+  type: :runtime
 description: Contains libraries that support defining, queueing, and executing jobs.
 email: daws23@gmail.com
 executables: []