qu-scheduler 0.0.1 → 0.1.0

data/MIT-LICENSE

@@ -1,4 +1,5 @@
-Copyright 2011 YOURNAME
+Copyright 2011 Morton Jonuschat
+Some parts copyright 2010 Ben VandenBos
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.md

@@ -0,0 +1,191 @@
+# Qu::Scheduler
+
+## Description
+
+qu-scheduler is an extension to [Qu](http://github.com/bkeepers/qu)
+that adds support for queueing jobs in the future.
+
+Currently qu-scheduler only works with qu-redis and requires Redis 2.0 or newer.
+
+Job scheduling is supported in two different ways: Recurring (scheduled) and
+Delayed.
+
+Scheduled jobs are like cron jobs, recurring on a regular basis. Delayed
+jobs are Qu jobs that you want to run at some point in the future.
+The syntax is pretty explanatory:
+
+    Qu.enqueue_in(5.days, SendFollowupEmail) # run a job in 5 days
+    # or
+    Qu.enqueue_at(5.days.from_now, SomeJob) # run SomeJob at a specific time
+
+## Installation
+
+    # Rails 3.x: add it to your Gemfile
+    gem 'qu-scheduler'
+
+There are just a single thing `qu-scheduler` needs to know about in order
+to do it's jobs: the schedule.  The easiest way to configure these things
+is via the rake task.  By default, `qu-scheduler` depends on the "qu:setup"
+rake task.  Since you probably already have this task, lets just put our
+configuration there. `qu-scheduler` pretty much inherits everything else
+from Qu.
+
+    # Qu tasks
+    require 'qu/tasks'
+    require 'qu-scheduler/tasks'
+
+    namespace :qu do
+      task :setup do
+        require 'qu'
+        require 'qu-scheduler'
+
+        # If you want to be able to dynamically change the schedule,
+        # uncomment this line.  A dynamic schedule can be updated via the
+        # Qu::Scheduler.set_schedule (and remove_schedule) methods.
+        # When dynamic is set to true, the scheduler process looks for
+        # schedule changes and applies them on the fly.
+        # Note: This feature is still under development
+        # Qu::Scheduler.dynamic = true
+
+        # The schedule doesn't need to be stored in a YAML, it just needs to
+        # be a hash.  YAML is usually the easiest.
+        Qu.schedule = YAML.load_file(Rails.root.join('config', 'your_resque_schedule.yml'))
+
+        # If your don't depend on your application environment you need to
+        # require your jobs here. Qu determines the queue exclusively from the
+        # class, so we need to have access to them.
+        require 'jobs'
+      end
+    end
+
+The scheduler process is just a rake task which is responsible for both
+queueing jobs from the schedule and polling the delayed queue for jobs
+ready to be pushed on to the work queues. For obvious reasons, this process
+never exits.
+
+    $ bundle exec rake qu:scheduler
+
+NOTE: You DO NOT want to run more than one instance of the scheduler.  Doing
+so will result in the same job being queued multiple times.  You only need one
+instance of the scheduler running per application, regardless of number of servers.
+
+If the scheduler process goes down for whatever reason, the delayed items
+that should have fired during the outage will fire once the scheduler process
+is started back up again (even if it is on a new machine). Missed scheduled
+jobs, however, will not fire upon recovery of the scheduler process.
+
+## Delayed jobs
+
+Delayed jobs are one-off jobs that you want to be put into a queue at some point
+in the future. The classic example is sending email:
+
+    Qu.enqueue_in(5.days, SendFollowUpEmail, current_user.id)
+
+This will store the job for 5 days in the Qu delayed queue at which time
+the scheduler process will pull it from the delayed queue and put it in the
+appropriate work queue for the given job. It will then be processed as soon as
+a worker is available (just like any other Qu job).
+
+NOTE: The job does not fire **exactly** at the time supplied. Rather, once that
+time is in the past, the job moves from the delayed queue to the actual work
+queue and will be completed as workers as free to process it.
+
+Also supported is `Qu.enqueue_at` which takes a timestamp to queue the job.
+
+The delayed queue is stored in redis and is persisted in the same way the
+standard Qu jobs are persisted (redis writing to disk). Delayed jobs differ
+from scheduled jobs in that if your scheduler process is down or workers are
+down when a particular job is supposed to be processed, they will simply "catch up"
+once they are started again.  Jobs are guaranteed to run (provided they make it
+into the delayed queue) after their given queue_at time has passed.
+
+Your jobs can specify one or more `before_schedule` and `after_schedule` hooks,
+to be run before or after scheduling. If *any* of your `before_schedule` hooks
+returns `false`, the job will *not* be scheduled and your `after_schedule` hooks
+will *not* be run.
+
+One other thing to note is that insertion into the delayed queue is O(log(n))
+since the jobs are stored in a redis sorted set (zset).  I can't imagine this
+being an issue for someone since redis is stupidly fast even at log(n), but full
+disclosure is always best.
+
+### Removing Delayed jobs
+
+If you have the need to cancel a delayed job, you can do it like this:
+
+    # after you've enqueued a job like:
+    Qu.enqueue_at(5.days.from_now, SendFollowUpEmail, current_user.id)
+    # remove the job with exactly the same parameters:
+    Qu.remove_delayed(SendFollowUpEmail, current_user.id)
+
+## Scheduled Jobs (Recurring Jobs)
+
+Scheduled (or recurring) jobs are logically no different than a standard cron
+job.  They are jobs that run based on a fixed schedule which is set at
+startup.
+
+The schedule is a list of job classes with arguments and a schedule frequency
+(in crontab syntax).  The schedule is just a hash, but is most likely stored in
+a YAML like this:
+
+    queue_documents_for_indexing:
+      cron: "0 0 * * *"
+      # you can use rufus-scheduler "every" syntax in place of cron if you prefer
+      # every: 1hr
+      klass: QueueDocuments
+      args:
+      description: "This job queues all content for indexing in solr"
+
+    clear_leaderboards_contributors:
+      cron: "30 6 * * 1"
+      klass: ClearLeaderboards
+      args: contributors
+      description: "This job resets the weekly leaderboard for contributions"
+
+NOTE: Six parameter cron's are also supported (as they supported by
+rufus-scheduler which powers the resque-scheduler process).  This allows you
+to schedule jobs per second (ie: "30 * * * * *" would fire a job every 30
+seconds past the minute).
+
+A big shout out to [rufus-scheduler](http://github.com/jmettraux/rufus-scheduler)
+for handling the heavy lifting of the actual scheduling engine.
+
+## Running in the background
+
+(Only supported with ruby >= 1.9). There are scenarios where it's helpful for
+the resque worker to run itself in the background (usually in combination with
+PIDFILE).  Use the BACKGROUND option so that rake will return as soon as the
+worker is started.
+
+    $ PIDFILE=./qu-scheduler.pid BACKGROUND=yes bundle exec rake qu:scheduler
+
+## Additional information
+
+by Ben VandenBos to the Qu queuing library.
+
+## Note on Patches / Pull Requests
+
+* Fork the project.
+* Make your feature addition or bug fix.
+* Add tests for it. This is important so I don't break it in a future version unintentionally.
+* Commit, do not mess with rakefile, version, or history.
+  (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
+* Send me a pull request. Bonus points for topic branches.
+
+## Credits
+
+This work is a port of [resque-scheduler](https://github.com/bvandenbos/resque-scheduler) by Ben VandenBos.
+Modified to work with the Qu queueing library by Morton Jonuschat.
+
+## Maintainers
+
+* [Morton Jonuschat](https://github.com/yabawock)
+
+## License
+
+MIT License
+
+## Copyright
+
+Copyright 2011 Morton Jonuschat
+Some parts copyright 2010 Ben VandenBos

data/lib/qu-scheduler.rb

@@ -1,4 +1 @@
-module Qu
-  module Scheduler
-  end
-end
+require 'qu/scheduler'

data/lib/qu-scheduler/tasks.rb

@@ -0,0 +1,35 @@
+require 'qu/tasks'
+
+namespace :qu do
+  desc "Start Qu Scheduler"
+  task :scheduler => :scheduler_setup do
+    require 'qu'
+    require 'qu-scheduler'
+
+    if ENV['BACKGROUND']
+      unless Process.respond_to?('daemon')
+        abort "Environment variable BACKGROUND is set, which requires ruby >= 1.9"
+      end
+      Process.daemon(true)
+    end
+
+    File.open(ENV['PIDFILE'], 'w') { |f| f << Process.pid.to_s } if ENV['PIDFILE']
+
+    Qu::Scheduler.dynamic = true if ENV['DYNAMIC_SCHEDULE']
+    Qu::Scheduler.run
+  end
+
+  task :scheduler_setup do
+    if ENV['INITIALIZER_PATH']
+      load ENV['INITIALIZER_PATH'].to_s.strip
+    else
+      Rake::Task['qu:setup'].invoke
+    end
+  end
+
+  # No-op task in case it doesn't already exist
+  task :setup
+end
+
+# Convenience tasks compatibility
+task 'resque:scheduler' => 'qu:scheduler'

data/lib/qu/extensions/scheduler.rb

@@ -0,0 +1,102 @@
+module Qu
+  module Extensions
+    module Scheduler
+      autoload :Redis, 'qu/extensions/scheduler/redis'
+
+      #
+      # Accepts a new schedule configuration of the form:
+      #
+      #   {'some_name' => {"cron" => "5/* * * *",
+      #                  "class" => "DoSomeWork",
+      #                  "args" => "work on this string",
+      #                  "description" => "this thing works it"s butter off"},
+      #    ...}
+      #
+      # 'some_name' can be anything and is used only to describe and reference
+      # the scheduled job
+      #
+      # :cron can be any cron scheduling string :job can be any resque job class
+      #
+      # :every can be used in lieu of :cron. see rufus-scheduler's 'every' usage
+      # for valid syntax. If :cron is present it will take precedence over :every.
+      #
+      # :class must be a resque worker class
+      #
+      # :args can be any yaml which will be converted to a ruby literal and
+      # passed in a params. (optional)
+      #
+      # :rails_envs is the list of envs where the job gets loaded. Envs are
+      # comma separated (optional)
+      #
+      # :description is just that, a description of the job (optional). If
+      # params is an array, each element in the array is passed as a separate
+      # param, otherwise params is passed in as the only parameter to perform.
+      def schedule=(schedule_hash)
+        if Qu::Scheduler.dynamic
+          schedule_hash.each do |name, job_spec|
+            backend.set_schedule(name, job_spec)
+          end
+        end
+        @schedule = schedule_hash
+      end
+
+      # Returns the schedule hash
+      def schedule
+        @schedule ||= {}
+      end
+
+      # reloads the schedule from redis
+      def reload_schedule!
+        @schedule = backend.get_schedules
+      end
+
+      # This method is nearly identical to +enqueue+ only it also
+      # takes a timestamp which will be used to schedule the job
+      # for queueing.  Until timestamp is in the past, the job will
+      # sit in the schedule list.
+      def enqueue_at(timestamp, klass, *args)
+        before_hooks = before_schedule_hooks(klass).collect do |hook|
+          klass.send(hook,*args)
+        end
+        return false if before_hooks.any? { |result| result == false }
+
+        backend.delayed_push(timestamp, Payload.new(:klass => klass, :args => args))
+
+        after_schedule_hooks(klass).collect do |hook|
+          klass.send(hook,*args)
+        end
+      end
+
+      # Identical to enqueue_at but takes number_of_seconds_from_now
+      # instead of a timestamp.
+      def enqueue_in(number_of_seconds_from_now, klass, *args)
+        enqueue_at(Time.now + number_of_seconds_from_now, klass, *args)
+      end
+
+      # Returns an array of delayed items for the given timestamp
+      def delayed_timestamp_peek(timestamp, start, count)
+        if 1 == count
+          r = backend.list_range "delayed:#{timestamp.to_i}", start, count
+          r.nil? ? [] : [r]
+        else
+          backend.list_range "delayed:#{timestamp.to_i}", start, count
+        end
+      end
+
+      private
+
+      def before_schedule_hooks(klass)
+        klass.methods.grep(/^before_schedule/).sort
+      end
+
+      def after_schedule_hooks(klass)
+        klass.methods.grep(/^after_schedule/).sort
+      end
+
+    end
+  end
+end
+Qu.send :extend, Qu::Extensions::Scheduler
+if defined? Qu::Backend::Redis
+  Qu::Backend::Redis.send :include, Qu::Extensions::Scheduler::Redis
+end

data/lib/qu/extensions/scheduler/redis.rb

@@ -0,0 +1,199 @@
+module Qu
+  module Extensions
+    module Scheduler
+      module Redis
+        # Does the dirty work of fetching a range of items from a Redis list
+        # and converting them into Ruby objects.
+        def list_range(key, start = 0, count = 1)
+          if count == 1
+            decode redis.lindex(key, start)
+          else
+            Array(redis.lrange(key, start, start+count-1)).map do |item|
+              decode item
+            end
+          end
+        end
+
+        # gets the schedule as it exists in redis
+        def get_schedules
+          if redis.exists(:schedules)
+            redis.hgetall(:schedules).tap do |h|
+              h.each do |name, config|
+                h[name] = decode(config)
+              end
+            end
+          else
+            nil
+          end
+        end
+
+        # Create or update a schedule with the provided name and configuration.
+        #
+        # Note: values for class and custom_job_class need to be strings,
+        # not constants.
+        #
+        #    Qu.set_schedule('some_job', { :class => 'SomeJob',
+        #                                  :every => '15mins',
+        #                                  :queue => 'high',
+        #                                  :args => '/tmp/poop' })
+        def set_schedule(name, config)
+          existing_config = get_schedule(name)
+          unless existing_config && existing_config == config
+            redis.hset(:schedules, name, encode(config))
+            redis.sadd(:schedules_changed, name)
+          end
+          config
+        end
+
+        # retrieve the schedule configuration for the given name
+        def get_schedule(name)
+          decode(redis.hget(:schedules, name))
+        end
+
+        # remove a given schedule by name
+        def remove_schedule(name)
+          redis.hdel(:schedules, name)
+          redis.sadd(:schedules_changed, name)
+        end
+
+        def update_schedule
+          if redis.scard(:schedules_changed) > 0
+            Qu::Scheduler.set_process_title "updating schedule"
+            Qu.reload_schedule!
+            while schedule_name = redis.spop(:schedules_changed)
+              if Qu.schedule.keys.include?(schedule_name)
+                Qu::Scheduler.unschedule_job(schedule_name)
+                Qu::Scheduler.load_schedule_job(schedule_name, Qu.schedule[schedule_name])
+              else
+                Qu::Scheduler.unschedule_job(schedule_name)
+              end
+            end
+            Qu::Scheduler.set_process_title "running"
+          end
+        end
+
+        # Pulls the schedule from Qu.schedule and loads it into the
+        # rufus scheduler instance
+        def load_schedule!
+          Qu::Scheduler.set_process_title "loading schedule"
+
+          # Need to load the schedule from redis for the first time if dynamic
+          Qu.reload_schedule! if Qu::Scheduler.dynamic
+
+          Qu.logger.warning("Schedule empty! Set Qu.schedule") if Qu.schedule.empty?
+
+          @@scheduled_jobs = {}
+
+          Qu.schedule.each do |name, config|
+            Qu::Scheduler.load_schedule_job(name, config)
+          end
+          redis.del(:schedules_changed)
+          Qu::Scheduler.set_process_title "running"
+        end
+
+        # Used internally to stuff the item into the schedule sorted list.
+        # +timestamp+ can be either in seconds or a datetime object
+        # Insertion if O(log(n)).
+        # Returns true if it's the first job to be scheduled at that time, else false
+        def delayed_push(timestamp, payload)
+          # First add this item to the list for this timestamp
+          redis.rpush("delayed:#{timestamp.to_i}", encode('klass' => payload.klass.to_s, 'args' => payload.args))
+
+          # Now, add this timestamp to the zsets.  The score and the value are
+          # the same since we'll be querying by timestamp, and we don't have
+          # anything else to store.
+          redis.zadd :delayed_queue_schedule, timestamp.to_i, timestamp.to_i
+        end
+
+        # Returns an array of timestamps based on start and count
+        def delayed_queue_peek(start, count)
+          Array(redis.zrange(:delayed_queue_schedule, start, start+count)).collect{|x| x.to_i}
+        end
+
+        # Returns the size of the delayed queue schedule
+        def delayed_queue_schedule_size
+          redis.zcard :delayed_queue_schedule
+        end
+
+        # Returns the number of jobs for a given timestamp in the delayed queue schedule
+        def delayed_timestamp_size(timestamp)
+          redis.llen("delayed:#{timestamp.to_i}").to_i
+        end
+
+        # Returns the next delayed queue timestamp
+        # (don't call directly)
+        def next_delayed_timestamp(at_time=nil)
+          items = redis.zrangebyscore :delayed_queue_schedule, '-inf', (at_time || Time.now).to_i, :limit => [0, 1]
+          timestamp = items.nil? ? nil : Array(items).first
+          timestamp.to_i unless timestamp.nil?
+        end
+
+        # Returns the next item to be processed for a given timestamp, nil if
+        # done. (don't call directly)
+        # +timestamp+ can either be in seconds or a datetime
+        def next_item_for_timestamp(timestamp)
+          key = "delayed:#{timestamp.to_i}"
+
+          item = decode redis.lpop(key)
+
+          # If the list is empty, remove it.
+          clean_up_timestamp(key, timestamp)
+          item
+        end
+
+        # Clears all jobs created with enqueue_at or enqueue_in
+        def reset_delayed_queue
+          Array(redis.zrange(:delayed_queue_schedule, 0, -1)).each do |item|
+            redis.del "delayed:#{item}"
+          end
+
+          redis.del :delayed_queue_schedule
+        end
+
+        # Given an encoded item, remove it from the delayed_queue
+        #
+        # This method is potentially very expensive since it needs to scan
+        # through the delayed queue for every timestamp.
+        def remove_delayed(klass, *args)
+          destroyed = 0
+          search = encode('klass' => klass.to_s, 'args' => args)
+          Array(redis.keys("delayed:*")).each do |key|
+            destroyed += redis.lrem key, 0, search
+          end
+          destroyed
+        end
+
+        # Given a timestamp and job (klass + args) it removes all instances and
+        # returns the count of jobs removed.
+        #
+        # O(N) where N is the number of jobs scheduled to fire at the given
+        # timestamp
+        def remove_delayed_job_from_timestamp(timestamp, klass, *args)
+          key = "delayed:#{timestamp.to_i}"
+          count = redis.lrem key, 0, encode('klass' => klass.to_s, 'args' => args)
+          clean_up_timestamp(key, timestamp)
+          count
+        end
+
+        def count_all_scheduled_jobs
+          total_jobs = 0
+          Array(redis.zrange(:delayed_queue_schedule, 0, -1)).each do |timestamp|
+            total_jobs += redis.llen("delayed:#{timestamp}").to_i
+          end
+          total_jobs
+        end
+
+        private
+
+        def clean_up_timestamp(key, timestamp)
+          # If the list is empty, remove it.
+          if 0 == redis.llen(key).to_i
+            redis.del key
+            redis.zrem :delayed_queue_schedule, timestamp.to_i
+          end
+        end
+      end
+    end
+  end
+end
+