brianjlandau-resque-scheduler 1.10.0

data/.gitignore

@@ -0,0 +1,3 @@
+pkg
+nbproject
+.rvmrc

data/HISTORY.md

@@ -0,0 +1,78 @@
+## 1.9.6 (2010-10-08)
+
+* Support for custom job classes (like resque-status) (mattetti)
+
+## 1.9.5 (2010-09-09)
+
+* Updated scheduler rake task to allow for an alternate setup task
+  to avoid loading the entire stack. (chewbranca)
+* Fixed sig issue on win32 (#25)
+
+## 1.9.4 (2010-07-29)
+
+* Adding ability to remove jobs from delayed queue (joshsz)
+* Fixing issue #23 (removing .present? reference)
+
+## 1.9.3 (2010-07-07)
+
+* Bug fix (#19)
+
+## 1.9.2 (2010-06-16)
+
+* Fixing issue with redis gem 2.0.1 and redis server 1.2.6 (dbackeus)
+
+## 1.9.1 (2010-06-04)
+
+* Fixing issue with redis server 1.2.6 and redis gem 2.0.1
+
+## 1.9.0 (2010-06-04)
+
+* Adding redis 2.0 support (bpo)
+
+## 1.8.2 (2010-06-04)
+
+* Adding queue now functionality to delayed timestamps (daviddoan)
+
+## 1.8.1 (2010-05-19)
+
+* Adding rails_env for scheduled jobs to support scoping jobs by
+  RAILS_ENV (gravis).
+* Fixing ruby 1.8.6 compatibility issue.
+* Adding gemspec for bundler support.
+
+## 1.8.0 (2010-04-14)
+
+* Moving version to match corresponding resque version
+* Sorting schedule on Scheduler tab
+* Adding tests for resque-web (gravis)
+
+## 1.0.5 (2010-03-01)
+
+* Fixed support for overriding queue from schedule config.
+* Removed resque-web dependency on loading the job classes for "Queue Now",
+  provided "queue" is specified in the schedule.
+* The queue is now stored with the job and arguments in the delayed queue so
+  there is no longer a need for the scheduler to load job classes to introspect
+  the queue.
+
+## 1.0.4 (2010-02-26)
+
+* Added support for specifying the queue to put the job onto. This allows for 
+  you to have one job that can go onto multiple queues and be able to schedule
+  jobs without having to load the job classes.
+
+## 1.0.3 (2010-02-11)
+
+* Added support for scheduled jobs with empty crons. This is helpful to have
+  jobs that you don't want on a schedule, but do want to be able to queue by
+  clicking a button.
+
+## 1.0.2 (2010-02-?)
+
+* Change Delayed Job tab to display job details if only 1 job exists
+  for a given timestamp
+
+## 1.0.1 (2010-01-?)
+
+* Bugfix: delayed jobs close together resulted in a 5 second sleep
+

data/LICENSE

@@ -0,0 +1,21 @@
+Copyright (c) 2010 Ben VandenBos
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+

data/README.markdown

@@ -0,0 +1,270 @@
+resque-scheduler
+===============
+
+Resque-scheduler is an extension to [Resque](http://github.com/defunkt/resque)
+that adds support for queueing items in the future.
+
+Requires redis >=1.1.
+
+
+Job scheduling is supported in two different way:
+
+### Recurring (scheduled)
+
+Recurring (or scheduled) jobs are logically no different than a standard cron
+job.  They are jobs that run based on a fixed schedule which is set at startup.
+
+The schedule is a list of Resque worker classes with arguments and a
+schedule frequency (in crontab syntax).  The schedule is just a hash, but
+is most likely stored in a YAML like so:
+
+    queue_documents_for_indexing:
+      cron: "0 0 * * *"
+      class: QueueDocuments
+      args: 
+      description: "This job queues all content for indexing in solr"
+
+    clear_leaderboards_contributors:
+      cron: "30 6 * * 1"
+      class: ClearLeaderboards
+      args: contributors
+      description: "This job resets the weekly leaderboard for contributions"
+
+A queue option can also be specified. When job will go onto the specified queue
+if it is available (Even if @queue is specified in the job class). When the
+queue is given it is not necessary for the scheduler to load the class.
+
+    clear_leaderboards_moderator:
+      cron: "30 6 * * 1"
+      class: ClearLeaderboards
+      queue: scoring
+      args: moderators
+      description: "This job resets the weekly leaderboard for moderators"
+
+And then set the schedule wherever you configure Resque, like so:
+
+    require 'resque_scheduler'
+    Resque.schedule = YAML.load_file(File.join(File.dirname(__FILE__), '../resque_schedule.yml'))
+
+Keep in mind, scheduled jobs behave like crons: if your scheduler process (more
+on that later) is not running when a particular job is supposed to be queued,
+it will NOT be ran later when the scheduler process is started back up.  In that
+sense, you can sort of think of the scheduler process as crond.  Delayed jobs,
+however, are different.
+
+A big shout out to [rufus-scheduler](http://github.com/jmettraux/rufus-scheduler)
+for handling the heavy lifting of the actual scheduling engine.
+
+### Delayed jobs
+
+Delayed jobs are one-off jobs that you want to be put into a queue at some point
+in the future.  The classic example is sending email:
+
+    Resque.enqueue_at(5.days.from_now, SendFollowUpEmail, :user_id => current_user.id)
+
+This will store the job for 5 days in the resque delayed queue at which time the
+scheduler process will pull it from the delayed queue and put it in the
+appropriate work queue for the given job and it will be processed as soon as
+a worker is available.
+
+NOTE: The job does not fire **exactly** at the time supplied.  Rather, once that
+time is in the past, the job moves from the delayed queue to the actual resque
+work queue and will be completed as workers as free to process it.
+
+Also supported is `Resque.enqueue_in` which takes an amount of time in seconds
+in which to queue the job.
+
+The delayed queue is stored in redis and is persisted in the same way the
+standard resque jobs are persisted (redis writing to disk). Delayed jobs differ
+from scheduled jobs in that if your scheduler process is down or workers are
+down when a particular job is supposed to be queue, they will simply "catch up"
+once they are started again.  Jobs are guaranteed to run (provided they make it
+into the delayed queue) after their given queue_at time has passed.
+
+One other thing to note is that insertion into the delayed queue is O(log(n))
+since the jobs are stored in a redis sorted set (zset).  I can't imagine this
+being an issue for someone since redis is stupidly fast even at log(n), but full
+disclosure is always best.
+
+*Removing Delayed jobs*
+
+If you have the need to cancel a delayed job, you can do so thusly:
+
+    # after you've enqueued a job like:
+    Resque.enqueue_at(5.days.from_now, SendFollowUpEmail, :user_id => current_user.id)
+    # remove the job with exactly the same parameters:
+    Resque.remove_delayed(SendFollowUpEmail, :user_id => current_user.id)
+
+### Schedule jobs per environment
+
+Resque-Scheduler allows to create schedule jobs for specific envs.  The arg
+`rails_env` (optional) can be used to determine which envs are concerned by the
+job:
+
+    create_fake_leaderboards:
+      cron: "30 6 * * 1"
+      class: CreateFakeLeaderboards
+      queue: scoring
+      args: 
+      rails_env: demo
+      description: "This job will auto-create leaderboards for our online demo"
+
+The scheduled job create_fake_leaderboards will be created only if the
+environment variable `RAILS_ENV` is set to demo:
+
+    $ RAILS_ENV=demo rake resque:scheduler 
+
+NOTE: If you have added the 2 lines bellow to your Rails Rakefile 
+(ie: lib/tasks/resque-scheduler.rake), the rails env is loaded automatically
+and you don't have to specify RAILS_ENV if the var is correctly set in
+environment.rb
+
+Alternatively, you can use your resque initializer to avoid loading the entire
+rails stack.
+
+    $ rake resque:scheduler INITIALIZER_PATH=config/initializers/resque.rb
+
+
+Multiple envs are allowed, separated by commas:
+
+    create_fake_leaderboards:
+      cron: "30 6 * * 1"
+      class: CreateFakeLeaderboards
+      queue: scoring
+      args: 
+      rails_env: demo, staging, production
+      description: "This job will auto-create leaderboards"
+
+NOTE: If you specify the `rails_env` arg without setting RAILS_ENV as an 
+environment variable, the job won't be loaded.
+
+### Dynamic Schedules
+
+If you need schedules that are defined based on user interactions inside of your application, this can be completed by loading it initially wherever you configure Resque and defining `Resque.reload_schedule!`:
+
+    module Resque
+      def self.reload_schedule!
+        self.schedule = MyScheduleModel.all.inject({}) {|schedule_hash, record| 
+          schedule_hash[record.name.to_sym] = record.attributes.select{|key, value| key != 'name' }
+          schedule_hash
+        }
+      end
+    end
+    
+    Resque.reload_schedule!
+
+To have the scheduler reload the schedule you just send it the `USR2` signal.
+
+### Support for customized Job classes
+
+Some Resque extensions like [resque-status](http://github.com/quirkey/resque-status) use custom job classes with a slightly different API signature.
+Resque-scheduler isn't trying to support all existing and future custom job classes, instead it supports a schedule flag so you can extend your custom class
+and make it support scheduled job.
+
+Let's pretend we have a JobWithStatus class called FakeLeaderboard
+
+		class FakeLeaderboard < Resque::JobWithStatus
+			def perfom
+				# do something and keep track of the status
+			end
+		end
+
+    create_fake_leaderboards:
+      cron: "30 6 * * 1"
+      queue: scoring
+      custom_job_class: FakeLeaderboard
+      args: 
+      rails_env: demo
+      description: "This job will auto-create leaderboards for our online demo and the status will update as the worker makes progress"
+
+If your extension doesn't support scheduled job, you would need to extend the custom job class to support the #scheduled method:
+
+    module Resque
+      class JobWithStatus
+        # Wrapper API to forward a Resque::Job creation API call into a JobWithStatus call.
+        def self.scheduled(queue, klass, *args)
+          create(args)
+        end
+      end
+    end
+
+
+Resque-web additions
+--------------------
+
+Resque-scheduler also adds to tabs to the resque-web UI.  One is for viewing
+(and manually queueing) the schedule and one is for viewing pending jobs in
+the delayed queue.
+
+The Schedule tab:
+
+![The Schedule Tab](http://img.skitch.com/20100111-km2f5gmtpbq23enpujbruj6mgk.png)
+
+The Delayed tab:
+
+![The Delayed Tab](http://img.skitch.com/20100111-ne4fcqtc5emkcuwc5qtais2kwx.jpg)
+
+Get get these to show up you need to pass a file to `resque-web` to tell it to
+include the `resque-scheduler` plugin.  You probably already have a file somewhere
+where you configure `resque`.  It probably looks something like this:
+
+    require 'resque' # include resque so we can configure it
+    Resque.redis = "redis_server:6379" # tell Resque where redis lives
+
+Now, you want to add the following:
+
+    require 'resque_scheduler' # include the resque_scheduler (this makes the tabs show up)
+
+And if you have a schedule you want to set, add this:
+
+    Resque.schedule = YAML.load_file(File.join(RAILS_ROOT, 'config/resque_schedule.yml')) # load the schedule
+
+Now make sure you're passing that file to resque-web like so:
+
+    resque-web ~/yourapp/config/resque_config.rb
+
+That should make the scheduler tabs show up in `resque-web`.
+
+
+
+Installation and the Scheduler process
+--------------------------------------
+
+To install:
+
+    gem install resque-scheduler
+
+You'll need to add this to your rakefile:
+
+    require 'resque_scheduler/tasks'
+    task "resque:setup" => :environment
+
+The scheduler process is just a rake task which is responsible for both queueing
+items from the schedule and polling the delayed queue for items ready to be
+pushed on to the work queues.  For obvious reasons, this process never exits.
+
+    $ rake resque:scheduler 
+
+Supported environment variables are `VERBOSE` and `MUTE`.  If either is set to
+any nonempty value, they will take effect.  `VERBOSE` simply dumps more output
+to stdout.  `MUTE` does the opposite and silences all output. `MUTE` supercedes
+`VERBOSE`.
+
+NOTE: You DO NOT want to run >1 instance of the scheduler.  Doing so will result
+in the same job being queued more than once.  You only need one instnace of the
+scheduler running per resque instance (regardless of number of machines).
+
+
+Plagurism alert
+---------------
+
+This was intended to be an extension to resque and so resulted in a lot of the
+code looking very similar to resque, particularly in resque-web and the views. I
+wanted it to be similar enough that someone familiar with resque could easily
+work on resque-scheduler.
+
+
+Contributing
+------------
+
+For bugs or suggestions, please just open an issue in github.

data/Rakefile

@@ -0,0 +1,48 @@
+load File.expand_path('tasks/resque_scheduler.rake')
+
+$LOAD_PATH.unshift 'lib'
+
+task :default => :test
+
+desc "Run tests"
+task :test do
+  Dir['test/*_test.rb'].each do |f|
+    require File.expand_path(f)
+  end
+end
+
+
+desc "Build a gem"
+task :gem => [ :test, :gemspec, :build ]
+
+begin
+  begin
+    require 'jeweler'
+  rescue LoadError
+    puts "Jeweler not available. Install it with: "
+    puts "gem install jeweler"
+  end
+
+  require 'resque_scheduler/version'
+
+  Jeweler::Tasks.new do |gemspec|
+    gemspec.name = "brianjlandau-resque-scheduler"
+    gemspec.summary = "Light weight job scheduling on top of Resque"
+    gemspec.description = %{Light weight job scheduling on top of Resque.
+  Adds methods enqueue_at/enqueue_in to schedule jobs in the future.
+  Also supports queueing jobs on a fixed, cron-like schedule.}
+    gemspec.email = "brianjlandau@gmail.com"
+    gemspec.homepage = "http://github.com/brianjlandau/resque-scheduler"
+    gemspec.authors = ["Ben VandenBos", "Brian Landau"]
+    gemspec.version = ResqueScheduler::Version
+
+    gemspec.add_dependency "redis", ">= 2.0.1"
+    gemspec.add_dependency "resque", ">= 1.8.0"
+    gemspec.add_dependency "rufus-scheduler"
+    gemspec.add_development_dependency "jeweler"
+    gemspec.add_development_dependency "mocha"
+    gemspec.add_development_dependency "rack-test"
+  end
+  
+  Jeweler::GemcutterTasks.new
+end

data/lib/resque/scheduler.rb

@@ -0,0 +1,195 @@
+require 'rufus/scheduler'
+require 'thwait'
+
+module Resque
+
+  class Scheduler
+
+    extend Resque::Helpers
+
+    class << self
+
+      # If true, logs more stuff...
+      attr_accessor :verbose
+      
+      # If set, produces no output
+      attr_accessor :mute
+
+      # Schedule all jobs and continually look for delayed jobs (never returns)
+      def run(da)
+
+        # trap signals
+        register_signal_handlers
+        
+        daemonize
+
+        # Load the schedule into rufus
+        load_schedule!
+
+        # Now start the scheduling part of the loop.
+        loop do
+          handle_delayed_items
+          poll_sleep
+        end
+
+        # never gets here.
+      end
+
+      # For all signals, set the shutdown flag and wait for current
+      # poll/enqueing to finish (should be almost istant).  In the
+      # case of sleeping, exit immediately.
+      def register_signal_handlers
+        trap("TERM") { shutdown }
+        trap("INT") { shutdown }
+        
+        begin
+          trap('QUIT') { shutdown   }
+          trap('USR1') { kill_child }
+          trap('USR2') { reload_schedule! }
+        rescue ArgumentError
+          warn "Signals QUIT and USR1 and USR2 not supported."
+        end
+      end
+      
+      def daemonize
+        Process.daemon(true)
+        if File.directory?('tmp/pids')
+          pid_file = File.expand_path('tmp/pids')
+          File.open(pid_file, 'w'){ |f| f.write(Process.pid) }
+          at_exit { File.delete(pid_file) if File.exist?(pid_file) }
+        elsif File.directory?('/usr/local/var/run')
+          pid_file = '/usr/local/var/run'
+          File.open(pid_file, 'w'){ |f| f.write(Process.pid) }
+          at_exit { File.delete(pid_file) if File.exist?(pid_file) }
+        end
+      end
+
+      # Pulls the schedule from Resque.schedule and loads it into the
+      # rufus scheduler instance
+      def load_schedule!
+        log! "Schedule empty! Set Resque.schedule" if Resque.schedule.empty?
+
+        Resque.schedule.each do |name, config|
+          # If rails_env is set in the config, enforce ENV['RAILS_ENV'] as
+          # required for the jobs to be scheduled.  If rails_env is missing, the
+          # job should be scheduled regardless of what ENV['RAILS_ENV'] is set
+          # to.
+          if config['rails_env'].nil? || rails_env_matches?(config)
+            log! "Scheduling #{name} "
+            if !config['cron'].nil? && config['cron'].length > 0
+              rufus_scheduler.cron config['cron'] do
+                log! "queuing #{config['class']} (#{name})"
+                enqueue_from_config(config)
+              end
+            else
+              log! "no cron found for #{config['class']} (#{name}) - skipping"
+            end
+          end
+        end
+      end
+
+      # Returns true if the given schedule config hash matches the current
+      # ENV['RAILS_ENV']
+      def rails_env_matches?(config)
+        config['rails_env'] && ENV['RAILS_ENV'] && config['rails_env'].gsub(/\s/,'').split(',').include?(ENV['RAILS_ENV'])
+      end
+
+      # Handles queueing delayed items
+      def handle_delayed_items
+        item = nil
+        begin
+          if timestamp = Resque.next_delayed_timestamp
+            enqueue_delayed_items_for_timestamp(timestamp)
+          end
+        # continue processing until there are no more ready timestamps
+        end while !timestamp.nil?
+      end
+      
+      # Enqueues all delayed jobs for a timestamp
+      def enqueue_delayed_items_for_timestamp(timestamp)
+        item = nil
+        begin
+          handle_shutdown do
+            if item = Resque.next_item_for_timestamp(timestamp)
+              log "queuing #{item['class']} [delayed]"
+              queue = item['queue'] || Resque.queue_from_class(constantize(item['class']))
+              # Support custom job classes like job with status
+              if (job_klass = item['custom_job_class']) && (job_klass != 'Resque::Job')
+                # custom job classes not supporting the same API calls must implement the #schedule method
+                constantize(job_klass).scheduled(queue, item['class'], *item['args'])
+              else
+                Resque::Job.create(queue, item['class'], *item['args'])
+              end
+            end
+          end
+        # continue processing until there are no more ready items in this timestamp
+        end while !item.nil?
+      end
+
+      def handle_shutdown
+        exit if @shutdown
+        yield
+        exit if @shutdown
+      end
+
+      # Enqueues a job based on a config hash
+      def enqueue_from_config(config)
+        args = config['args'] || config[:args]
+        klass_name = config['class'] || config[:class]
+        params = args.nil? ? [] : Array(args)
+        queue = config['queue'] || config[:queue] || Resque.queue_from_class(constantize(klass_name))
+        # Support custom job classes like job with status
+        if (job_klass = config['custom_job_class']) && (job_klass != 'Resque::Job')
+          # custom job classes not supporting the same API calls must implement the #schedule method
+          constantize(job_klass).scheduled(queue, klass_name, *params)
+        else
+          Resque::Job.create(queue, klass_name, *params)
+        end        
+      end
+
+      def rufus_scheduler
+        @rufus_scheduler ||= Rufus::Scheduler.start_new
+      end
+
+      # Stops old rufus scheduler and creates a new one.  Returns the new
+      # rufus scheduler
+      def clear_schedule!
+        rufus_scheduler.stop
+        @rufus_scheduler = nil
+        rufus_scheduler
+      end
+      
+      def reload_schedule!
+        clear_schedule!
+        Resque.reload_schedule!
+        load_schedule!
+      end
+
+      # Sleeps and returns true
+      def poll_sleep
+        @sleeping = true
+        handle_shutdown { sleep 5 }
+        @sleeping = false
+        true
+      end
+
+      # Sets the shutdown flag, exits if sleeping
+      def shutdown
+        @shutdown = true
+        exit if @sleeping
+      end
+
+      def log!(msg)
+        puts "#{Time.now.strftime("%Y-%m-%d %H:%M:%S")} #{msg}" unless mute
+      end
+
+      def log(msg)
+        # add "verbose" logic later
+        log!(msg) if verbose
+      end
+
+    end
+
+  end
+
+end