nulogy-resque-scheduler 1.10.15

data/HISTORY.md

@@ -0,0 +1,78 @@
+## 1.9.6 (2010-10-08)
+
+* Support for custom job classes (like resque-status) (mattetti)
+
+## 1.9.5 (2010-09-09)
+
+* Updated scheduler rake task to allow for an alternate setup task
+  to avoid loading the entire stack. (chewbranca)
+* Fixed sig issue on win32 (#25)
+
+## 1.9.4 (2010-07-29)
+
+* Adding ability to remove jobs from delayed queue (joshsz)
+* Fixing issue #23 (removing .present? reference)
+
+## 1.9.3 (2010-07-07)
+
+* Bug fix (#19)
+
+## 1.9.2 (2010-06-16)
+
+* Fixing issue with redis gem 2.0.1 and redis server 1.2.6 (dbackeus)
+
+## 1.9.1 (2010-06-04)
+
+* Fixing issue with redis server 1.2.6 and redis gem 2.0.1
+
+## 1.9.0 (2010-06-04)
+
+* Adding redis 2.0 support (bpo)
+
+## 1.8.2 (2010-06-04)
+
+* Adding queue now functionality to delayed timestamps (daviddoan)
+
+## 1.8.1 (2010-05-19)
+
+* Adding rails_env for scheduled jobs to support scoping jobs by
+  RAILS_ENV (gravis).
+* Fixing ruby 1.8.6 compatibility issue.
+* Adding gemspec for bundler support.
+
+## 1.8.0 (2010-04-14)
+
+* Moving version to match corresponding resque version
+* Sorting schedule on Scheduler tab
+* Adding tests for resque-web (gravis)
+
+## 1.0.5 (2010-03-01)
+
+* Fixed support for overriding queue from schedule config.
+* Removed resque-web dependency on loading the job classes for "Queue Now",
+  provided "queue" is specified in the schedule.
+* The queue is now stored with the job and arguments in the delayed queue so
+  there is no longer a need for the scheduler to load job classes to introspect
+  the queue.
+
+## 1.0.4 (2010-02-26)
+
+* Added support for specifying the queue to put the job onto. This allows for 
+  you to have one job that can go onto multiple queues and be able to schedule
+  jobs without having to load the job classes.
+
+## 1.0.3 (2010-02-11)
+
+* Added support for scheduled jobs with empty crons. This is helpful to have
+  jobs that you don't want on a schedule, but do want to be able to queue by
+  clicking a button.
+
+## 1.0.2 (2010-02-?)
+
+* Change Delayed Job tab to display job details if only 1 job exists
+  for a given timestamp
+
+## 1.0.1 (2010-01-?)
+
+* Bugfix: delayed jobs close together resulted in a 5 second sleep
+

data/LICENSE

@@ -0,0 +1,21 @@
+Copyright (c) 2010 Ben VandenBos
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+

data/README.markdown

@@ -0,0 +1,275 @@
+resque-scheduler
+===============
+
+Resque-scheduler is an extension to [Resque](http://github.com/defunkt/resque)
+that adds support for queueing items in the future.
+
+Requires redis >=1.1.
+
+
+Job scheduling is supported in two different way:
+
+### Recurring (scheduled)
+
+Recurring (or scheduled) jobs are logically no different than a standard cron
+job.  They are jobs that run based on a fixed schedule which is set at startup.
+
+The schedule is a list of Resque worker classes with arguments and a
+schedule frequency (in crontab syntax).  The schedule is just a hash, but
+is most likely stored in a YAML like so:
+
+    queue_documents_for_indexing:
+      cron: "0 0 * * *"
+      class: QueueDocuments
+      args: 
+      description: "This job queues all content for indexing in solr"
+
+    clear_leaderboards_contributors:
+      cron: "30 6 * * 1"
+      class: ClearLeaderboards
+      args: contributors
+      description: "This job resets the weekly leaderboard for contributions"
+
+A queue option can also be specified. When job will go onto the specified queue
+if it is available (Even if @queue is specified in the job class). When the
+queue is given it is not necessary for the scheduler to load the class.
+
+    clear_leaderboards_moderator:
+      cron: "30 6 * * 1"
+      class: ClearLeaderboards
+      queue: scoring
+      args: moderators
+      description: "This job resets the weekly leaderboard for moderators"
+
+And then set the schedule wherever you configure Resque, like so:
+
+    require 'resque_scheduler'
+    Resque.schedule = YAML.load_file(File.join(File.dirname(__FILE__), '../resque_schedule.yml'))
+
+Keep in mind, scheduled jobs behave like crons: if your scheduler process (more
+on that later) is not running when a particular job is supposed to be queued,
+it will NOT be ran later when the scheduler process is started back up.  In that
+sense, you can sort of think of the scheduler process as crond.  Delayed jobs,
+however, are different.
+
+A big shout out to [rufus-scheduler](http://github.com/jmettraux/rufus-scheduler)
+for handling the heavy lifting of the actual scheduling engine.
+
+### Delayed jobs
+
+Delayed jobs are one-off jobs that you want to be put into a queue at some point
+in the future.  The classic example is sending email:
+
+    Resque.enqueue_at(5.days.from_now, SendFollowUpEmail, :user_id => current_user.id)
+
+This will store the job for 5 days in the resque delayed queue at which time the
+scheduler process will pull it from the delayed queue and put it in the
+appropriate work queue for the given job and it will be processed as soon as
+a worker is available.
+
+NOTE: The job does not fire **exactly** at the time supplied.  Rather, once that
+time is in the past, the job moves from the delayed queue to the actual resque
+work queue and will be completed as workers as free to process it.
+
+Also supported is `Resque.enqueue_in` which takes an amount of time in seconds
+in which to queue the job.
+
+The delayed queue is stored in redis and is persisted in the same way the
+standard resque jobs are persisted (redis writing to disk). Delayed jobs differ
+from scheduled jobs in that if your scheduler process is down or workers are
+down when a particular job is supposed to be queue, they will simply "catch up"
+once they are started again.  Jobs are guaranteed to run (provided they make it
+into the delayed queue) after their given queue_at time has passed.
+
+One other thing to note is that insertion into the delayed queue is O(log(n))
+since the jobs are stored in a redis sorted set (zset).  I can't imagine this
+being an issue for someone since redis is stupidly fast even at log(n), but full
+disclosure is always best.
+
+*Removing Delayed jobs*
+
+If you have the need to cancel a delayed job, you can do so thusly:
+
+    # after you've enqueued a job like:
+    Resque.enqueue_at(5.days.from_now, SendFollowUpEmail, :user_id => current_user.id)
+    # remove the job with exactly the same parameters:
+    Resque.remove_delayed(SendFollowUpEmail, :user_id => current_user.id)
+
+### Schedule jobs per environment
+
+Resque-Scheduler allows to create schedule jobs for specific envs.  The arg
+`rails_env` (optional) can be used to determine which envs are concerned by the
+job:
+
+    create_fake_leaderboards:
+      cron: "30 6 * * 1"
+      class: CreateFakeLeaderboards
+      queue: scoring
+      args: 
+      rails_env: demo
+      description: "This job will auto-create leaderboards for our online demo"
+
+The scheduled job create_fake_leaderboards will be created only if the
+environment variable `RAILS_ENV` is set to demo:
+
+    $ RAILS_ENV=demo rake resque:scheduler 
+
+NOTE: If you have added the 2 lines bellow to your Rails Rakefile 
+(ie: lib/tasks/resque-scheduler.rake), the rails env is loaded automatically
+and you don't have to specify RAILS_ENV if the var is correctly set in
+environment.rb
+
+Alternatively, you can use your resque initializer to avoid loading the entire
+rails stack.
+
+    $ rake resque:scheduler INITIALIZER_PATH=config/initializers/resque.rb
+
+
+Multiple envs are allowed, separated by commas:
+
+    create_fake_leaderboards:
+      cron: "30 6 * * 1"
+      class: CreateFakeLeaderboards
+      queue: scoring
+      args: 
+      rails_env: demo, staging, production
+      description: "This job will auto-create leaderboards"
+
+NOTE: If you specify the `rails_env` arg without setting RAILS_ENV as an 
+environment variable, the job won't be loaded.
+
+### Dynamic Schedules
+
+If needed you can also have schedules that are dynamically defined and updated inside of your application. This can be completed by loading the schedule initially wherever you configure Resque and setting `Resque::Scheduler.dynamic` to `true`. Then subsequently updating the "`schedules`" key in redis, namespaced to the Resque namespace. The "`schedules`" key is expected to be a redis hash data type, where the key is the name of the schedule and the value is a JSON encoded hash of the schedule configuration.
+
+When the scheduler loops it will look for differences between the existing schedule and the current schedule in redis. If there are differences it will make the necessary changes to the running schedule.
+
+To force the scheduler to reload the schedule you just send it the `USR2` signal.
+
+Convenience methods are provided to add/update, delete, and retrieve individual schedule items from the `schedules` in redis:
+
+* `Resque.set_schedule(name, config)`
+* `Resque.get_schedule(name)`
+* `Resque.remove_schedule(name)`
+
+For example:
+
+    Resque.set_schedule("create_fake_leaderboards", {
+      :cron => "30 6 * * 1",
+      :class => "CreateFakeLeaderboards",
+      :queue => scoring
+    })
+
+### Support for customized Job classes
+
+Some Resque extensions like [resque-status](http://github.com/quirkey/resque-status) use custom job classes with a slightly different API signature.
+Resque-scheduler isn't trying to support all existing and future custom job classes, instead it supports a schedule flag so you can extend your custom class
+and make it support scheduled job.
+
+Let's pretend we have a JobWithStatus class called FakeLeaderboard
+
+		class FakeLeaderboard < Resque::JobWithStatus
+			def perfom
+				# do something and keep track of the status
+			end
+		end
+
+    create_fake_leaderboards:
+      cron: "30 6 * * 1"
+      queue: scoring
+      custom_job_class: FakeLeaderboard
+      args: 
+      rails_env: demo
+      description: "This job will auto-create leaderboards for our online demo and the status will update as the worker makes progress"
+
+If your extension doesn't support scheduled job, you would need to extend the custom job class to support the #scheduled method:
+
+    module Resque
+      class JobWithStatus
+        # Wrapper API to forward a Resque::Job creation API call into a JobWithStatus call.
+        def self.scheduled(queue, klass, *args)
+          create(args)
+        end
+      end
+    end
+
+
+Resque-web additions
+--------------------
+
+Resque-scheduler also adds to tabs to the resque-web UI.  One is for viewing
+(and manually queueing) the schedule and one is for viewing pending jobs in
+the delayed queue.
+
+The Schedule tab:
+
+![The Schedule Tab](http://img.skitch.com/20100111-km2f5gmtpbq23enpujbruj6mgk.png)
+
+The Delayed tab:
+
+![The Delayed Tab](http://img.skitch.com/20100111-ne4fcqtc5emkcuwc5qtais2kwx.jpg)
+
+Get get these to show up you need to pass a file to `resque-web` to tell it to
+include the `resque-scheduler` plugin.  You probably already have a file somewhere
+where you configure `resque`.  It probably looks something like this:
+
+    require 'resque' # include resque so we can configure it
+    Resque.redis = "redis_server:6379" # tell Resque where redis lives
+
+Now, you want to add the following:
+
+    require 'resque_scheduler' # include the resque_scheduler (this makes the tabs show up)
+
+And if you have a schedule you want to set, add this:
+
+    Resque.schedule = YAML.load_file(File.join(RAILS_ROOT, 'config/resque_schedule.yml')) # load the schedule
+
+Now make sure you're passing that file to resque-web like so:
+
+    resque-web ~/yourapp/config/resque_config.rb
+
+That should make the scheduler tabs show up in `resque-web`.
+
+
+
+Installation and the Scheduler process
+--------------------------------------
+
+To install:
+
+    gem install resque-scheduler
+
+You'll need to add this to your rakefile:
+
+    require 'resque_scheduler/tasks'
+    task "resque:setup" => :environment
+
+The scheduler process is just a rake task which is responsible for both queueing
+items from the schedule and polling the delayed queue for items ready to be
+pushed on to the work queues.  For obvious reasons, this process never exits.
+
+    $ rake resque:scheduler 
+
+Supported environment variables are `VERBOSE` and `MUTE`.  If either is set to
+any nonempty value, they will take effect.  `VERBOSE` simply dumps more output
+to stdout.  `MUTE` does the opposite and silences all output. `MUTE` supercedes
+`VERBOSE`.
+
+NOTE: You DO NOT want to run >1 instance of the scheduler.  Doing so will result
+in the same job being queued more than once.  You only need one instnace of the
+scheduler running per resque instance (regardless of number of machines).
+
+
+Plagurism alert
+---------------
+
+This was intended to be an extension to resque and so resulted in a lot of the
+code looking very similar to resque, particularly in resque-web and the views. I
+wanted it to be similar enough that someone familiar with resque could easily
+work on resque-scheduler.
+
+
+Contributing
+------------
+
+For bugs or suggestions, please just open an issue in github.

data/Rakefile

@@ -0,0 +1,49 @@
+load File.expand_path('tasks/resque_scheduler.rake')
+
+$LOAD_PATH.unshift 'lib'
+
+task :default => :test
+
+desc "Run tests"
+task :test do
+  Dir['test/*_test.rb'].each do |f|
+    require File.expand_path(f)
+  end
+end
+
+
+desc "Build a gem"
+task :gem => [ :test, :gemspec, :build ]
+
+begin
+  begin
+    require 'jeweler'
+  rescue LoadError
+    puts "Jeweler not available. Install it with: "
+    puts "gem install jeweler"
+  end
+
+  require 'resque_scheduler/version'
+
+  Jeweler::Tasks.new do |gemspec|
+    gemspec.name = "nulogy-resque-scheduler"
+    gemspec.summary = "Light weight job scheduling on top of Resque"
+    gemspec.description = %{Light weight job scheduling on top of Resque.
+  Adds methods enqueue_at/enqueue_in to schedule jobs in the future.
+  Also supports queueing jobs on a fixed, cron-like schedule.}
+    gemspec.email = "engineering@nulogy.com"
+    gemspec.homepage = "http://github.com/nulogy/resque-scheduler"
+    gemspec.authors = ["Ben VandenBos", "Brian Landau", "Sean Kirby", "Tanzeeb Khalili", "Justin Fitzsimmons"]
+    gemspec.version = ResqueScheduler::Version
+
+    gemspec.add_dependency "redis", ">= 2.0.1"
+    gemspec.add_dependency "resque", ">= 1.8.0"
+    gemspec.add_dependency "tanzeeb-rufus-scheduler"
+    gemspec.add_dependency "airbrake"
+    gemspec.add_development_dependency "jeweler"
+    gemspec.add_development_dependency "mocha"
+    gemspec.add_development_dependency "rack-test"
+  end
+
+  Jeweler::GemcutterTasks.new
+end

data/lib/resque/scheduler.rb

@@ -0,0 +1,256 @@
+require 'rufus/scheduler'
+require 'airbrake'
+require 'thwait'
+
+module Resque
+
+  class Scheduler
+
+    extend Resque::Helpers
+
+    class << self
+
+      # If true, logs more stuff...
+      attr_accessor :verbose
+
+      # If set, produces no output
+      attr_accessor :mute
+
+      # If set, will try to update the schulde in the loop
+      attr_accessor :dynamic
+
+      # If set, will tell rufus to use the airbrake exception handler
+      attr_accessor :airbrake
+
+      # the Rufus::Scheduler jobs that are scheduled
+      def scheduled_jobs
+        @@scheduled_jobs
+      end
+
+      # Schedule all jobs and continually look for delayed jobs (never returns)
+      def run
+        $0 = "resque-scheduler: Starting"
+        # trap signals
+        register_signal_handlers
+
+        # Load the schedule into rufus
+        procline "Loading Schedule"
+        load_schedule!
+
+        # Now start the scheduling part of the loop.
+        loop do
+          handle_delayed_items
+          update_schedule if dynamic
+          poll_sleep
+        end
+
+        # never gets here.
+      end
+
+      # For all signals, set the shutdown flag and wait for current
+      # poll/enqueing to finish (should be almost istant).  In the
+      # case of sleeping, exit immediately.
+      def register_signal_handlers
+        trap("TERM") { shutdown }
+        trap("INT") { shutdown }
+
+        begin
+          trap('QUIT') { shutdown   }
+          trap('USR1') { kill_child }
+          trap('USR2') { reload_schedule! }
+        rescue ArgumentError
+          warn "Signals QUIT and USR1 and USR2 not supported."
+        end
+      end
+
+      # Pulls the schedule from Resque.schedule and loads it into the
+      # rufus scheduler instance
+      def load_schedule!
+        log! "Schedule empty! Set Resque.schedule" if Resque.schedule.empty?
+
+        @@scheduled_jobs = {}
+
+        Resque.schedule.each do |name, config|
+          load_schedule_job(name, config)
+        end
+        Resque.redis.del(:schedules_changed)
+        procline "Schedules Loaded"
+      end
+
+      # Loads a job schedule into the Rufus::Scheduler and stores it in @@scheduled_jobs
+      def load_schedule_job(name, config)
+        # If rails_env is set in the config, enforce ENV['RAILS_ENV'] as
+        # required for the jobs to be scheduled.  If rails_env is missing, the
+        # job should be scheduled regardless of what ENV['RAILS_ENV'] is set
+        # to.
+        if config['rails_env'].nil? || rails_env_matches?(config)
+          log! "Scheduling #{name} "
+          interval_defined = false
+          interval_types = %w{cron every}
+          interval_types.each do |interval_type|
+            if !config[interval_type].nil? && config[interval_type].length > 0
+              begin
+                @@scheduled_jobs[name] = rufus_scheduler.send(interval_type, config[interval_type]) do
+                  log! "queueing #{config['class']} (#{name})"
+                  enqueue_from_config(config)
+                end
+              rescue Exception => e
+                log! "#{e.class.name}: #{e.message}"
+              end
+              interval_defined = true
+              break
+            end
+          end
+          unless interval_defined
+            log! "no #{interval_types.join(' / ')} found for #{config['class']} (#{name}) - skipping"
+          end
+        end
+      end
+
+      # Returns true if the given schedule config hash matches the current
+      # ENV['RAILS_ENV']
+      def rails_env_matches?(config)
+        config['rails_env'] && ENV['RAILS_ENV'] && config['rails_env'].gsub(/\s/,'').split(',').include?(ENV['RAILS_ENV'])
+      end
+
+      # Handles queueing delayed items
+      def handle_delayed_items(at_time = nil)
+        if timestamp = Resque.next_delayed_timestamp(at_time)
+          procline "Processing Delayed Items"
+          while !timestamp.nil?
+            enqueue_delayed_items_for_timestamp(timestamp)
+            timestamp = Resque.next_delayed_timestamp(at_time)
+          end
+        end
+      end
+
+      # Enqueues all delayed jobs for a timestamp
+      def enqueue_delayed_items_for_timestamp(timestamp)
+        item = nil
+        begin
+          handle_shutdown do
+            if item = Resque.next_item_for_timestamp(timestamp)
+              log "queuing #{item['class']} [delayed]"
+              klass = constantize(item['class'])
+              queue = item['queue'] || Resque.queue_from_class(klass)
+              # Support custom job classes like job with status
+              if (job_klass = item['custom_job_class']) && (job_klass != 'Resque::Job')
+                # custom job classes not supporting the same API calls must implement the #schedule method
+                constantize(job_klass).scheduled(queue, item['class'], *item['args'])
+              else
+                Resque.enqueue_to(queue, klass, *item['args'])
+                #Resque::Job.create(queue, klass, *item['args'])
+              end
+            end
+          end
+        # continue processing until there are no more ready items in this timestamp
+        end while !item.nil?
+      end
+
+      def handle_shutdown
+        exit if @shutdown
+        yield
+        exit if @shutdown
+      end
+
+      # Enqueues a job based on a config hash
+      def enqueue_from_config(config)
+        args = config['args'] || config[:args]
+        klass_name = config['class'] || config[:class]
+        klass = constantize(klass_name)
+        params = args.nil? ? [] : Array(args)
+        queue = config['queue'] || config[:queue] || Resque.queue_from_class(klass)
+        # Support custom job classes like job with status
+        if (job_klass = config['custom_job_class']) && (job_klass != 'Resque::Job')
+          # custom job classes not supporting the same API calls must implement the #schedule method
+          constantize(job_klass).scheduled(queue, klass_name, *params)
+        else
+          Resque.enqueue_to(queue, klass, *params)
+        end
+      end
+
+      def rufus_scheduler
+        @rufus_scheduler ||= Rufus::Scheduler.start_new
+        if self.airbrake
+          def @rufus_scheduler.handle_exception(job, exception)
+            puts "Exception caught, notifying Airbrake"
+            id = Airbrake.notify(exception)
+            puts "Airbrake id: #{id}"
+          end
+        end
+        @rufus_scheduler
+      end
+
+      # Stops old rufus scheduler and creates a new one.  Returns the new
+      # rufus scheduler
+      def clear_schedule!
+        rufus_scheduler.stop
+        @rufus_scheduler = nil
+        @@scheduled_jobs = {}
+        rufus_scheduler
+      end
+
+      def reload_schedule!
+        procline "Reloading Schedule"
+        clear_schedule!
+        Resque.reload_schedule!
+        load_schedule!
+      end
+
+      def update_schedule
+        if Resque.redis.scard(:schedules_changed) > 0
+          procline "Updating schedule"
+          Resque.reload_schedule!
+          while schedule_name = Resque.redis.spop(:schedules_changed)
+            if Resque.schedule.keys.include?(schedule_name)
+              unschedule_job(schedule_name)
+              load_schedule_job(schedule_name, Resque.schedule[schedule_name])
+            else
+              unschedule_job(schedule_name)
+            end
+          end
+        end
+        procline "Schedules Loaded"
+      end
+
+      def unschedule_job(name)
+        if scheduled_jobs[name]
+          log "Removing schedule #{name}"
+          scheduled_jobs[name].unschedule
+          @@scheduled_jobs.delete(name)
+        end
+      end
+
+      # Sleeps and returns true
+      def poll_sleep
+        @sleeping = true
+        handle_shutdown { sleep 5 }
+        @sleeping = false
+        true
+      end
+
+      # Sets the shutdown flag, exits if sleeping
+      def shutdown
+        @shutdown = true
+        exit if @sleeping
+      end
+
+      def log!(msg)
+        puts "#{Time.now.strftime("%Y-%m-%d %H:%M:%S")} #{msg}" unless mute
+      end
+
+      def log(msg)
+        # add "verbose" logic later
+        log!(msg) if verbose
+      end
+
+      def procline(string)
+        $0 = "resque-scheduler-#{ResqueScheduler::Version}: #{string}"
+        log! $0
+      end
+
+    end
+
+  end
+
+end