delayed_job_tracer 0.9.0

data/.gitignore

@@ -0,0 +1,4 @@
+*.gem
+.bundle
+Gemfile.lock
+pkg/*

data/Gemfile

@@ -0,0 +1,2 @@
+source :rubygems
+gemspec

data/README.md

@@ -0,0 +1,111 @@
+delayed_job_tracer -- Like a tracer bullet for your delayed_job queue.
+====================================
+
+## DESCRIPTION
+
+The delayed_job_tracer gem is designed to monitor the delayed_job gem ([https://github.com/collectiveidea/delayed_job](https://github.com/collectiveidea/delayed_job)).  It will alert you via e-mail if something goes wrong, such as if the delayed_job process crashes or one of its jobs fails or takes too long to complete.
+
+This gem also detects wither e-mails sent through delayed_job are actually being delivered by sending test messages (like tracer bullets) to an e-mail account then checking that account to ensure they were processed and dispatched via the delayed_job queue.  The e-mail delivery detection allows you to be notified if e-mails can't be sent out either because of delayed_job having choked or because of an issue with the e-mail account the application uses to send e-mail.
+
+
+## Primary Features
+
+* Sends an e-mail notification when delayed_job crashes or is unable to complete a job in a reasonable amount of time.
+* Sends an e-mail notification when the Rails application is unable to successfully deliver e-mails via delayed_job.
+* Circumvents the need for a process monitoring tool like Monit or God for delayed_job processes.
+* Keeps memory usage low by working outside of Rails.
+
+
+## Rails and delayed_job support
+
+* The master branch and all 1.x series gems work with delayed_job 2.1.x and above (Rails 3.x)
+* The 2-3-stable branch and all 0.9.x series gems work with delayed_job 2.0.7 and above (Rails 2.3)
+
+
+## Why use a Cron'd ruby process to monitor delayed_job and e-mail delivery?
+
+* Cron is often considered to be more reliable than process monitoring tools like Monit or God.
+* The Cron'd Ruby process only uses a modest amount of memory (~27MB) for a few seconds at regular intervals, whereas a rake task or runner would equate to the entire size of the Rails application, with a memory footprint that grows over time.
+
+
+## Why use a Perl Script to send alert e-mails to admins?
+
+* It's fast
+* It doesn't rely on the Rails application
+* It doesn't rely on the delayed_job process you're monitoring with this gem
+
+
+## Prerequisites
+
+1) Install XML and SSL Packages:
+
+Note: These are for Ubuntu.  You may need to find alternate packages for your platform.
+
+  sudo apt-get install libxml2-dev libxslt-dev libnet-ssleay-perl libcrypt-ssleay-perl libio-socket-ssl-perl
+
+2) sendEmail Perl script:
+
+Download from "http://caspian.dotconf.net/menu/Software/SendEmail":http://caspian.dotconf.net/menu/Software/SendEmail and follow the setup instructions, or just follow these instructions:
+
+  wget -c http://caspian.dotconf.net/menu/Software/SendEmail/sendEmail-v1.56.tar.gz
+  tar xvfz sendEmail-v1.56.tar.gz
+  sudo cp -a sendEmail-v1.56/sendEmail /usr/local/bin
+  chmod +x /usr/local/bin/sendEmail
+
+Run it to be sure it works:
+
+  sendEmail
+
+Cleanup:
+
+  rm -rf sendEmail-v1.56*
+
+
+## Installation
+
+1) Add the gem to your Gemfile:
+
+  gem "delayed_job_tracer"
+
+2) Generate the config file:
+
+  rails g delayed_job_tracer
+
+## Configuration
+
+1) Create a 'DelayedJobTracer' folder/label in the e-mail account you configure the plugin to send test messages to and a filter to move all incoming messages with '[DelayedJobTracer]' in the subject line to this folder (or label if you're using Gmail).  Set the name of this folder in the config/delayed_job_tracer_config.yml file - as it will be the folder that the Ruby process checks for test messages.
+
+2) Add a Cron job on the server:
+
+Example for running every 15 minutes, replace with your actual app and ruby locations:
+
+  */15 * * * * /usr/local/bin/delayed_job_tracer -c /opt/apps/appname/current/config/delayed_job_tracer_config.yml
+
+Note: If you're using bundler to install gems within an application-specific gem environment, you may need to also install the gem manually on the server so that the executable is available via the location in the example cron entry above.
+
+Finish configuring your settings in the config/delayed_job_tracer_config.yml file.  Your database and ActionMailer config will be applied via the generator, but you'll still need to provide an alternate e-mail account to send administrative notifications through, provide the admin's e-mail address and so forth.
+
+
+## License
+
+The MIT License
+
+Copyright (c) 2010-2011 Kenny Johnston
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/Rakefile

@@ -0,0 +1,2 @@
+require 'bundler'
+Bundler::GemHelper.install_tasks

data/bin/delayed_job_tracer

@@ -0,0 +1,38 @@
+#!/usr/bin/env ruby
+
+require 'delayed_job_tracer'
+require 'optparse'
+require 'yaml'
+
+options = {}
+
+optparse = OptionParser.new do |opts|
+  # Displayed at top of help screen
+  opts.banner = "Usage: delayed_job_tracer -c /opt/apps/myapp/current/config/delayed_job_tracer_config.yml"
+
+  opts.on( '-c', '--config FILE', 'Path to config file' ) do |file|
+    options[:config] = file
+  end
+
+  opts.on( '-h', '--help', 'Display this screen' ) do
+    puts opts
+    exit
+  end
+end
+
+# Parse the command-line. Remember there are two forms
+# of the parse method. The 'parse' method simply parses
+# ARGV, while the 'parse!' method parses ARGV and removes
+# any options found there, as well as any parameters for
+# the options. What's left is the list of files to resize.
+optparse.parse!
+
+begin
+  raise "no config file provided" unless options[:config]
+  raise "no config file found at '#{options[:config]}'" unless File.exists?(options[:config])
+  config = YAML.load_file(options[:config])
+  raise "incorrect config file: '#{options[:config]}'" unless config['app']
+  DelayedJobTracer.run(config)
+rescue Exception => e
+  puts "Error: #{e}"
+end

data/delayed_job_tracer.gemspec

@@ -0,0 +1,26 @@
+# -*- encoding: utf-8 -*-
+$:.push File.expand_path("../lib", __FILE__)
+require "delayed_job_tracer/version"
+
+Gem::Specification.new do |s|
+  s.name        = "delayed_job_tracer"
+  s.version     = DelayedJobTracer::VERSION
+  s.platform    = Gem::Platform::RUBY
+  s.authors     = ["Kenny Johnston"]
+  s.email       = ["kjohnston.ca@gmail.com"]
+  s.homepage    = "https://github.com/kjohnston/delayed_job_tracer"
+  s.summary     = %q{Like a tracer bullet for your delayed_job queue}
+  s.description = %q{Monitors the delayed_job queue and periodically tests its ability to deliver
+                     e-mail messages and e-mails you if something goes wrong, such as the delayed_job
+                     process crashes or one of its jobs fails or takes too long to complete.}
+
+  s.add_runtime_dependency "delayed_job", "~> 2.0.7"
+  s.add_runtime_dependency "mms2r",       "2.4.1"
+  s.add_runtime_dependency "mysql2",      ">= 0.2.6", "<= 0.4"
+  s.add_runtime_dependency "tmail",       "1.2.7.1"
+
+  s.files              = `git ls-files`.split("\n")
+  s.test_files         = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.executables        = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  s.require_paths      = ["lib"]
+end

data/generators/delayed_job_tracer/delayed_job_tracer_generator.rb

@@ -0,0 +1,25 @@
+class DelayedJobTracerGenerator < Rails::Generator::Base
+
+  attr_accessor :app_name, :db_host, :db_database, :db_username, :db_password,
+                :mail_domain, :mail_username, :mail_password
+
+  def manifest
+    @app_name      = Rails.root.to_s.split('/').last
+
+    db_config      = Rails::Configuration.new
+    @db_host       = db_config.database_configuration["production"]["host"]
+    @db_database   = db_config.database_configuration["production"]["database"]
+    @db_username   = db_config.database_configuration["production"]["username"]
+    @db_password   = db_config.database_configuration["production"]["password"]
+
+    mail_config    = ActionMailer::Base.smtp_settings
+    @mail_domain   = mail_config[:address]
+    @mail_username = mail_config[:user_name]
+    @mail_password = mail_config[:password]
+
+    record do |m|
+      m.template 'delayed_job_tracer_config.yml', File.join('config', 'delayed_job_tracer_config.yml')
+    end
+  end
+
+end

data/generators/delayed_job_tracer/templates/delayed_job_tracer_config.yml

@@ -0,0 +1,55 @@
+# Application's name
+# This will be the prefix of alert e-mail subject lines, which will have brackets
+# added around it.
+# Example:  App Name => [App Name]
+app:
+  name: <%= app_name %>
+
+
+# Administrative e-mail address to send alerts to
+admin:
+  email:
+
+
+# Number of seconds after a delayed_job is considered stale.
+# An e-mail will be sent to the admin if records older than this are found
+# in the delayed_jobs table.  This will catch 3 scenarios:
+#   1) The delayed_job process locked up or crashed and jobs stopeed running
+#   2) One or more jobs failed and could not succeed on retry
+#   3) One or more jobs are taking longer than the number of seconds set here
+#      to complete
+delayed_job:
+  stale: 3600
+
+
+# Application's production database
+database:
+  ip: <%= db_host %>
+  database: <%= db_database %>
+  user: <%= db_username %>
+  password: <%= db_password %>
+
+
+# E-mail account to monitor for test messages
+# This account will be checked periodically via a cron'd execution of runner.rb to
+# ensure that a recent monitoring message has been delivered sucessfully to that
+# e-mail account.
+monitor:
+  server: <%= mail_domain %>
+  username: <%= mail_username %>
+  password: <%= mail_password %>
+  port: 993
+  ssl: true
+  folder: DelayedJobTracer
+
+
+# E-mail account to send admin alerts through via a perl script.
+# You should use a different account than the applicaiton uses, so alert messages can
+# still be sent if for example the Gmail send limit is exceeded for the application's
+# e-mail account and Google puts it on lockdown.
+alert:
+  server: smtp.gmail.com
+  username:
+  password:
+  port: 25
+  tls: true

data/lib/delayed_job_tracer.rb

@@ -0,0 +1,26 @@
+if defined?(ActionMailer)
+  require File.dirname(__FILE__) + '/delayed_job_tracer/delayed_job_tracer_mailer.rb'
+else
+  require File.dirname(__FILE__) + '/delayed_job_tracer/notifier'
+  require File.dirname(__FILE__) + '/delayed_job_tracer/message_finder'
+  require File.dirname(__FILE__) + '/delayed_job_tracer/mysql_interface'
+
+  class DelayedJobTracer
+
+    def self.config
+      @@config
+    end
+
+    def self.config=(c)
+      @@config = c
+    end
+
+    def self.run(config)
+      @@config = config
+      Notifier.notify_admin_of_email_issue unless MessageFinder.found_recent_message?
+      Notifier.notify_admin_of_queue_issue unless MySQLInterface.delayed_job_queue_ok?
+      MySQLInterface.queue_delayed_job
+    end
+
+  end
+end

data/lib/delayed_job_tracer/delayed_job_tracer_mailer.rb

@@ -0,0 +1,16 @@
+class DelayedJobTracerMailer < ActionMailer::Base
+
+  def delayed_job_tracer_test_message
+    config      = YAML.load_file(Rails.root.join('config', 'delayed_job_tracer_config.yml'))
+    @from       = ActionMailer::Base.smtp_settings[:user_name]
+    @recipients = config["monitor"]["username"]
+    @subject    = "[DelayedJobTracer] #{Time.zone.now}"
+    @sent_on    = Time.zone.now
+    @body       = "This is a test message to ensure messages are being delivered successfully via delayed_job."
+  end
+
+  def self.enqueue_delayed_job_tracer_test_message
+    delay.deliver_delayed_job_tracer_test_message
+  end
+
+end

data/lib/delayed_job_tracer/message_finder.rb

@@ -0,0 +1,50 @@
+require 'rubygems'
+require 'net/imap'
+require 'tmail'
+require 'mms2r'
+require 'yaml'
+
+class MessageFinder
+
+  def self.found_recent_message?
+    c = DelayedJobTracer.config['monitor']
+
+    server   = c['server']
+    port     = c['port'] || 143
+    ssl      = c['ssl'] || false
+    username = c['username']
+    password = c['password']
+    folder   = c['folder']
+
+    imap = Net::IMAP.new server, port, ssl
+    imap.login username, password
+    imap.select folder
+    uids = imap.search(["UNSEEN"])
+
+    emails = []
+    times = []
+
+    uids.each do |uid|
+      mdata = imap.fetch(uid, 'RFC822')[0].attr['RFC822']
+      tmail = TMail::Mail.parse mdata
+      mms   = MMS2R::Media.new tmail
+      emails << {:mdata => mdata, :tmail => tmail, :mms => mms, :uid => uid}
+    end
+
+    emails.each do |email|
+      # Collects the dates of unread messages
+      times << email[:tmail].date
+      # Marks unread messages as read
+      imap.store email[:uid], "+FLAGS", [:Seen]
+    end
+
+    imap.disconnect
+
+    # Delete tmp files used by mms2r
+    emails.map{|m| m[:mms].purge }
+
+    # Just checks to see if there was at least one recent unread message
+    return true unless times.blank?
+  end
+
+end

data/lib/delayed_job_tracer/mysql_interface.rb

@@ -0,0 +1,52 @@
+require 'mysql2'
+
+class MySQLInterface
+
+  # Connects to the db and submits a query
+  def self.query(sql)
+    c = DelayedJobTracer.config['database']
+    d = Mysql2::Client.new(:host => c['ip'], :database => c['database'], :username => c['user'], :password => c['password'])
+    d.query(sql)
+  end
+
+  # Returns true if there are no stale records
+  def self.delayed_job_queue_ok?
+    query(delayed_job_stale_records).count.zero?
+  end
+
+  # Inserts a delayed_job record
+  def self.queue_delayed_job
+    query(delayed_job_record).to_s
+  end
+
+  # SQL for selecting stale delayed_job records
+  def self.delayed_job_stale_records
+    c = DelayedJobTracer.config['delayed_job']
+    "SELECT * FROM delayed_jobs WHERE created_at < '#{(Time.now-c['stale']).utc.strftime("%Y-%m-%d %H:%M:%S")}'"
+  end
+
+  # SQL for inserting a delayed_job record
+  def self.delayed_job_record
+    "INSERT INTO delayed_jobs (`handler`, `run_at`, `created_at`, `updated_at`) VALUES
+      ('#{delayed_job_handler}', '#{mysql_timestamp}', '#{mysql_timestamp}', '#{mysql_timestamp}')"
+  end
+
+  # SQL helper method for inserting a delayed_job record
+  def self.delayed_job_handler
+    "--- !ruby/struct:Delayed::PerformableMethod
+object: LOAD;DelayedJobTracerMailer
+method: :deliver_delayed_job_tracer_test_message
+args: []"
+  end
+
+  # Timestamp in the format that e-mails expect
+  def self.email_timestamp
+    Time.now.strftime("%a, %e %b %Y %H:%M:%S %z")
+  end
+
+  # Timestamp in the format that MySQL expects
+  def self.mysql_timestamp
+    Time.now.utc.strftime("%Y-%m-%d %H:%M:%S")
+  end
+
+end

data/lib/delayed_job_tracer/notifier.rb

@@ -0,0 +1,29 @@
+require 'yaml'
+
+class Notifier
+
+  def self.notify_admin_of_email_issue
+    subject_suffix = 'E-mail Issue'
+    message_suffix = 'is having trouble sending e-mail via delayed_job, please investigate.'
+    send_notification(subject_suffix, message_suffix)
+  end
+
+  def self.notify_admin_of_queue_issue
+    subject_suffix = 'Queue Issue'
+    message_suffix = 'has one or more stale delayed_jobs, please investigate.'
+    send_notification(subject_suffix, message_suffix)
+  end
+
+  def self.send_notification(subject_suffix, message_suffix)
+    c         = DelayedJobTracer.config
+    account   = c['alert']
+    recipient = c['admin']['email']
+    appname   = c['app']['name']
+    subject   = "[#{appname}] DelayedJobTracer: " + subject_suffix
+    message   = "#{appname} " + message_suffix
+    system    "/usr/local/bin/sendEmail -f #{account['username']} -t #{recipient} -u '#{subject}' -m #{message} \
+                -s #{account['server']}:#{account['port']} -xu #{account['username']} -xp #{account['password']} \
+                #{ '-o tls=yes' if c['alert']['tls'] == 'true' }"
+  end
+
+end

data/lib/delayed_job_tracer/version.rb

@@ -0,0 +1,3 @@
+module DelayedJobTracer
+  VERSION = "0.9.0"
+end

data/rails/init.rb

@@ -0,0 +1 @@
+require "delayed_job_tracer"

metadata

@@ -0,0 +1,154 @@
+--- !ruby/object:Gem::Specification 
+name: delayed_job_tracer
+version: !ruby/object:Gem::Version 
+  hash: 59
+  prerelease: 
+  segments: 
+  - 0
+  - 9
+  - 0
+  version: 0.9.0
+platform: ruby
+authors: 
+- Kenny Johnston
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2011-09-13 00:00:00 Z
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: delayed_job
+  prerelease: false
+  requirement: &id001 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        hash: 1
+        segments: 
+        - 2
+        - 0
+        - 7
+        version: 2.0.7
+  type: :runtime
+  version_requirements: *id001
+- !ruby/object:Gem::Dependency 
+  name: mms2r
+  prerelease: false
+  requirement: &id002 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - "="
+      - !ruby/object:Gem::Version 
+        hash: 29
+        segments: 
+        - 2
+        - 4
+        - 1
+        version: 2.4.1
+  type: :runtime
+  version_requirements: *id002
+- !ruby/object:Gem::Dependency 
+  name: mysql2
+  prerelease: false
+  requirement: &id003 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 27
+        segments: 
+        - 0
+        - 2
+        - 6
+        version: 0.2.6
+    - - <=
+      - !ruby/object:Gem::Version 
+        hash: 3
+        segments: 
+        - 0
+        - 4
+        version: "0.4"
+  type: :runtime
+  version_requirements: *id003
+- !ruby/object:Gem::Dependency 
+  name: tmail
+  prerelease: false
+  requirement: &id004 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - "="
+      - !ruby/object:Gem::Version 
+        hash: 81
+        segments: 
+        - 1
+        - 2
+        - 7
+        - 1
+        version: 1.2.7.1
+  type: :runtime
+  version_requirements: *id004
+description: |-
+  Monitors the delayed_job queue and periodically tests its ability to deliver
+                       e-mail messages and e-mails you if something goes wrong, such as the delayed_job
+                       process crashes or one of its jobs fails or takes too long to complete.
+email: 
+- kjohnston.ca@gmail.com
+executables: 
+- delayed_job_tracer
+extensions: []
+
+extra_rdoc_files: []
+
+files: 
+- .gitignore
+- Gemfile
+- README.md
+- Rakefile
+- bin/delayed_job_tracer
+- delayed_job_tracer.gemspec
+- generators/delayed_job_tracer/delayed_job_tracer_generator.rb
+- generators/delayed_job_tracer/templates/delayed_job_tracer_config.yml
+- lib/delayed_job_tracer.rb
+- lib/delayed_job_tracer/delayed_job_tracer_mailer.rb
+- lib/delayed_job_tracer/message_finder.rb
+- lib/delayed_job_tracer/mysql_interface.rb
+- lib/delayed_job_tracer/notifier.rb
+- lib/delayed_job_tracer/version.rb
+- rails/init.rb
+homepage: https://github.com/kjohnston/delayed_job_tracer
+licenses: []
+
+post_install_message: 
+rdoc_options: []
+
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.8.6
+signing_key: 
+specification_version: 3
+summary: Like a tracer bullet for your delayed_job queue
+test_files: []
+