resque-scheduler 4.2.1 → 4.5.0

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,74 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, gender identity and expression, level of experience,
+nationality, personal appearance, race, religion, or sexual identity and
+orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at resque@librelist.com. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at [http://contributor-covenant.org/version/1/4][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/4/

data/Gemfile

@@ -1,4 +1,12 @@
 # vim:fileencoding=utf-8
 source 'https://rubygems.org'
 
+case req = ENV['RESQUE']
+when nil
+when 'master'
+  gem 'resque', git: 'https://github.com/resque/resque'
+else
+  gem 'resque', req
+end
+
 gemspec

data/README.md

@@ -1,17 +1,17 @@
 resque-scheduler
 ================
 
-[![Dependency Status](https://gemnasium.com/resque/resque-scheduler.png)](https://gemnasium.com/resque/resque-scheduler)
-[![Gem Version](https://badge.fury.io/rb/resque-scheduler.png)](http://badge.fury.io/rb/resque-scheduler)
-[![Build Status](https://travis-ci.org/resque/resque-scheduler.png?branch=master)](https://travis-ci.org/resque/resque-scheduler)
-[![Code Climate](https://codeclimate.com/github/resque/resque-scheduler.png)](https://codeclimate.com/github/resque/resque-scheduler)
+
+[![Gem Version](https://badge.fury.io/rb/resque-scheduler.svg)](https://badge.fury.io/rb/resque-scheduler)
+[![Ruby specs](https://github.com/resque/resque-scheduler/actions/workflows/ruby.yml/badge.svg)](https://github.com/resque/resque-scheduler/actions)
+[![Code Climate](https://codeclimate.com/github/resque/resque-scheduler/badges/gpa.svg)](https://codeclimate.com/github/resque/resque-scheduler)
 
 ### Description
 
 Resque-scheduler is an extension to [Resque](http://github.com/resque/resque)
 that adds support for queueing items in the future.
 
-Job scheduling is supported in two different way: Recurring (scheduled) and
+Job scheduling is supported in two different ways: Recurring (scheduled) and
 Delayed.
 
 Scheduled jobs are like cron jobs, recurring on a regular basis.  Delayed
@@ -158,7 +158,7 @@ following task to wherever tasks are kept, such as
 ```ruby
 task 'resque:pool:setup' do
   Resque::Pool.after_prefork do |job|
-    Resque.redis.client.reconnect
+    Resque.redis._client.reconnect
   end
 end
 ```
@@ -233,6 +233,17 @@ Resque.remove_delayed_selection { |args| args[0]['account_id'] == current_accoun
 Resque.remove_delayed_selection { |args| args[0]['user_id'] == current_user.id }
 ```
 
+If you need to cancel a delayed job based on some matching arguments AND by which class the job is, but don't wish to specify each argument from when the job was created, you can do like so:
+
+``` ruby
+# after you've enqueued a job like:
+Resque.enqueue_at(5.days.from_now, SendFollowUpEmail, :account_id => current_account.id, :user_id => current_user.id)
+# remove jobs matching just the account and that were of the class SendFollowUpEmail:
+Resque.remove_delayed_selection(SendFollowUpEmail) { |args| args[0]['account_id'] == current_account.id }
+# or remove jobs matching just the user and that were of the class SendFollowUpEmail:
+Resque.remove_delayed_selection(SendFollowUpEmail) { |args| args[0]['user_id'] == current_user.id }
+```
+
 If you need to enqueue immediately a delayed job based on some matching arguments, but don't wish to specify each argument from when the job was created, you can do like so:
 
 ``` ruby
@@ -289,6 +300,19 @@ clear_leaderboards_contributors:
   description: "This job resets the weekly leaderboard for contributions"
 ```
 
+If you would like to setup a job that is executed manually you can configure like this in your YAML file.
+
+```yaml
+ImportOrdersManual:
+  description: 'Import Amazon Orders Manual'
+  custom_job_class: 'AmazonMws::ImportOrdersJob'
+  never: "* * * * *"
+  queue: high
+  description: "This is a manual job for importing orders."
+  args:
+    days_in_arrears: 7
+```
+
 The queue value is optional, but if left unspecified resque-scheduler will
 attempt to get the queue from the job class, which means it needs to be
 defined.  If you're getting "uninitialized constant" errors, you probably
@@ -323,8 +347,8 @@ for handling the heavy lifting of the actual scheduling engine.
 #### Dynamic schedules
 
 Dynamic schedules are programmatically set on a running `resque-scheduler`.
-All [rufus-scheduler](http://github.com/jmettraux/rufus-scheduler) options are supported
-when setting schedules.
+Most [rufus-scheduler](http://github.com/jmettraux/rufus-scheduler) options are supported
+when setting schedules. Specifically the `overlap` option will not work.
 
 Dynamic schedules are not enabled by default. To be able to dynamically set schedules, you
 must pass the following to `resque-scheduler` initialization (see *Installation* above for a more complete example):
@@ -501,7 +525,7 @@ RESQUE_SCHEDULER_MASTER_LOCK_PREFIX=MyApp: rake resque:scheduler
 
 ### resque-web Additions
 
-Resque-scheduler also adds to tabs to the resque-web UI.  One is for viewing
+Resque-scheduler also adds two tabs to the resque-web UI.  One is for viewing
 (and manually queueing) the schedule and one is for viewing pending jobs in
 the delayed queue.
 
@@ -535,11 +559,7 @@ require 'resque/scheduler/server'
 
 That should make the scheduler tabs show up in `resque-web`.
 
-#### Changes as of 2.0.0
-
-As of resque-scheduler 2.0.0, it's no longer necessary to have the resque-web
-process aware of the schedule because it reads it from redis.  But prior to
-2.0, you'll want to make sure you load the schedule in this file as well.
+You'll want to make sure you load the schedule in this file as well.
 Something like this:
 
 ```ruby
@@ -608,18 +628,18 @@ with resque could easily work on resque-scheduler.
 
 Working on resque-scheduler requires the following:
 
-* A relatively modern Ruby interpreter (MRI 1.9+ is what's tested)
+* A relatively modern Ruby interpreter
 * bundler
 
 The development setup looks like this, which is roughly the same thing
-that happens on Travis CI:
+that happens on Travis CI and Appveyor:
 
 ``` bash
 # Install everything
 bundle install
 
 # Make sure tests are green before you change stuff
-bundle exec rake
+bundle exec rubocop && bundle exec rake
 # Change stuff
 # Repeat
 ```

data/Rakefile

@@ -1,13 +1,9 @@
 # vim:fileencoding=utf-8
 require 'bundler/gem_tasks'
 require 'rake/testtask'
-require 'rubocop/rake_task'
 require 'yard'
 
-task default: [:rubocop, :test] unless RUBY_PLATFORM =~ /java/
-task default: [:test] if RUBY_PLATFORM =~ /java/
-
-RuboCop::RakeTask.new
+task default: :test
 
 Rake::TestTask.new do |t|
   t.libs << 'test'

data/lib/resque/scheduler/cli.rb

@@ -128,6 +128,7 @@ module Resque
       def option_parser
         OptionParser.new do |opts|
           opts.banner = BANNER
+          opts.version = Resque::Scheduler::VERSION
           OPTIONS.each do |opt|
             opts.on(*opt[:args], &opt[:callback].call(options))
           end

data/lib/resque/scheduler/delaying_extensions.rb

@@ -24,7 +24,7 @@ module Resque
       def enqueue_at_with_queue(queue, timestamp, klass, *args)
         return false unless plugin.run_before_schedule_hooks(klass, *args)
 
-        if Resque.inline? || timestamp.to_i < Time.now.to_i
+        if Resque.inline? || timestamp.to_i <= Time.now.to_i
           # Just create the job and let resque perform it right away with
           # inline.  If the class is a custom job class, call self#scheduled
           # on it. This allows you to do things like
@@ -33,7 +33,7 @@ module Resque
           if klass.respond_to?(:scheduled)
             klass.scheduled(queue, klass.to_s, *args)
           else
-            Resque::Job.create(queue, klass, *args)
+            Resque.enqueue_to(queue, klass, *args)
           end
         else
           delayed_push(timestamp, job_to_hash_with_queue(queue, klass, args))
@@ -45,6 +45,9 @@ module Resque
       # Identical to enqueue_at but takes number_of_seconds_from_now
       # instead of a timestamp.
       def enqueue_in(number_of_seconds_from_now, klass, *args)
+        unless number_of_seconds_from_now.is_a?(Numeric)
+          raise ArgumentError, 'Please supply a numeric number of seconds'
+        end
         enqueue_at(Time.now + number_of_seconds_from_now, klass, *args)
       end
 
@@ -53,14 +56,17 @@ module Resque
       # number of seconds has passed.
       def enqueue_in_with_queue(queue, number_of_seconds_from_now,
                                 klass, *args)
+        unless number_of_seconds_from_now.is_a?(Numeric)
+          raise ArgumentError, 'Please supply a numeric number of seconds'
+        end
         enqueue_at_with_queue(queue, Time.now + number_of_seconds_from_now,
                               klass, *args)
       end
 
       # Used internally to stuff the item into the schedule sorted list.
-      # +timestamp+ can be either in seconds or a datetime object Insertion
-      # if O(log(n)).  Returns true if it's the first job to be scheduled at
-      # that time, else false
+      # +timestamp+ can be either in seconds or a datetime object. The
+      # insertion time complexity is O(log(n)). Returns true if it's
+      # the first job to be scheduled at that time, else false.
       def delayed_push(timestamp, item)
         # First add this item to the list for this timestamp
         redis.rpush("delayed:#{timestamp.to_i}", encode(item))
@@ -82,6 +88,7 @@ module Resque
       end
 
       # Returns the size of the delayed queue schedule
+      # this does not represent the number of items in the queue to be scheduled
       def delayed_queue_schedule_size
         redis.zcard :delayed_queue_schedule
       end
@@ -143,6 +150,11 @@ module Resque
         remove_delayed_job(search)
       end
 
+      def remove_delayed_in_queue(klass, queue, *args)
+        search = encode(job_to_hash_with_queue(queue, klass, args))
+        remove_delayed_job(search)
+      end
+
       # Given an encoded item, enqueue it now
       def enqueue_delayed(klass, *args)
         hash = job_to_hash(klass, args)
@@ -151,6 +163,13 @@ module Resque
         end
       end
 
+      def enqueue_delayed_with_queue(klass, queue, *args)
+        hash = job_to_hash_with_queue(queue, klass, args)
+        remove_delayed_in_queue(klass, queue, *args).times do
+          Resque::Scheduler.enqueue_from_config(hash)
+        end
+      end
+
       # Given a block, remove jobs that return true from a block
       #
       # This allows for removal of delayed jobs that have arguments matching
@@ -175,7 +194,15 @@ module Resque
         found_jobs.reduce(0) do |sum, encoded_job|
           decoded_job = decode(encoded_job)
           klass = Util.constantize(decoded_job['class'])
-          sum + enqueue_delayed(klass, *decoded_job['args'])
+          queue = decoded_job['queue']
+
+          if queue
+            jobs_queued = enqueue_delayed_with_queue(klass, queue, *decoded_job['args'])
+          else
+            jobs_queued = enqueue_delayed(klass, *decoded_job['args'])
+          end
+
+          jobs_queued + sum
         end
       end
 
@@ -265,6 +292,8 @@ module Resque
         { class: klass.to_s, args: args, queue: queue }
       end
 
+      # Removes a job from the queue, but not modify the timestamp schedule. This method
+      # will not effect the output of `delayed_queue_schedule_size`
       def remove_delayed_job(encoded_job)
         return 0 if Resque.inline?
 
@@ -277,6 +306,9 @@ module Resque
           end
         end
 
+        # timestamp key is not removed from the schedule, this is done later
+        # by the scheduler loop
+
         return 0 if replies.nil? || replies.empty?
         replies.each_slice(2).map(&:first).inject(:+)
       end

data/lib/resque/scheduler/env.rb

@@ -32,14 +32,18 @@ module Resque
 
         # Need to set this here for conditional Process.daemon redirect of
         # stderr/stdout to /dev/null
-        Resque::Scheduler.quiet = !!options[:quiet]
+        Resque::Scheduler.quiet = if options.key?(:quiet)
+                                    !!options[:quiet]
+                                  else
+                                    true
+                                  end
 
         unless Process.respond_to?('daemon')
           abort 'background option is set, which requires ruby >= 1.9'
         end
 
         Process.daemon(true, !Resque::Scheduler.quiet)
-        Resque.redis.client.reconnect
+        Resque.redis._client.reconnect
       end
 
       def setup_pid_file
@@ -60,13 +64,13 @@ module Resque
 
           c.dynamic = !!options[:dynamic] if options.key?(:dynamic)
 
-          c.env = options[:env] if options.key(:env)
+          c.env = options[:env] if options.key?(:env)
 
           c.logfile = options[:logfile] if options.key?(:logfile)
 
           c.logformat = options[:logformat] if options.key?(:logformat)
 
-          if psleep = options[:poll_sleep_amount] && !psleep.nil?
+          if (psleep = options[:poll_sleep_amount]) && !psleep.nil?
             c.poll_sleep_amount = Float(psleep)
           end
 

data/lib/resque/scheduler/lock/resilient.rb

@@ -6,19 +6,11 @@ module Resque
     module Lock
       class Resilient < Base
         def acquire!
-          Resque.redis.evalsha(
-            acquire_sha,
-            keys: [key],
-            argv: [value]
-          ).to_i == 1
+          evalsha(:acquire, [key], [value]).to_i == 1
         end
 
         def locked?
-          Resque.redis.evalsha(
-            locked_sha,
-            keys: [key],
-            argv: [value]
-          ).to_i == 1
+          evalsha(:locked, [key], [value]).to_i == 1
         end
 
         def timeout=(seconds)
@@ -32,11 +24,26 @@ module Resque
 
         private
 
+        def evalsha(script, keys, argv, refresh: false)
+          sha_method_name = "#{script}_sha"
+          Resque.redis.evalsha(
+            send(sha_method_name, refresh),
+            keys: keys,
+            argv: argv
+          )
+        rescue Redis::CommandError => e
+          if e.message =~ /NOSCRIPT/
+            refresh = true
+            retry
+          end
+          raise
+        end
+
         def locked_sha(refresh = false)
           @locked_sha = nil if refresh
 
           @locked_sha ||=
-            Resque.redis.script(:load, <<-EOF.gsub(/^ {14}/, ''))
+            Resque.data_store.redis.script(:load, <<-EOF.gsub(/^ {14}/, ''))
               if redis.call('GET', KEYS[1]) == ARGV[1]
               then
                 redis.call('EXPIRE', KEYS[1], #{timeout})
@@ -55,7 +62,7 @@ module Resque
           @acquire_sha = nil if refresh
 
           @acquire_sha ||=
-            Resque.redis.script(:load, <<-EOF.gsub(/^ {14}/, ''))
+            Resque.data_store.redis.script(:load, <<-EOF.gsub(/^ {14}/, ''))
               if redis.call('SETNX', KEYS[1], ARGV[1]) == 1
               then
                 redis.call('EXPIRE', KEYS[1], #{timeout})

data/lib/resque/scheduler/locking.rb

@@ -76,7 +76,7 @@ module Resque
 
       def release_master_lock
         master_lock.release
-      rescue Errno::EAGAIN, Errno::ECONNRESET, Redis::CannotConnectError
+      rescue *INTERMITTENT_ERRORS
         @master_lock = nil
       end
 
@@ -97,7 +97,7 @@ module Resque
       end
 
       def redis_master_version
-        Resque.redis.info['redis_version'].to_f
+        Resque.data_store.redis.info['redis_version'].to_f
       end
     end
   end

data/lib/resque/scheduler/scheduling_extensions.rb

@@ -36,7 +36,7 @@ module Resque
       # :args can be any yaml which will be converted to a ruby literal and
       # passed in a params. (optional)
       #
-      # :rails_envs is the list of envs where the job gets loaded. Envs are
+      # :rails_env is the list of envs where the job gets loaded. Envs are
       # comma separated (optional)
       #
       # :description is just that, a description of the job (optional). If
@@ -101,12 +101,13 @@ module Resque
       end
 
       # remove a given schedule by name
-      def remove_schedule(name)
+      # Preventing a reload is optional and available to batch operations
+      def remove_schedule(name, reload = true)
         non_persistent_schedules.delete(name)
         redis.hdel(:persistent_schedules, name)
         redis.sadd(:schedules_changed, name)
 
-        reload_schedule!
+        reload_schedule! if reload
       end
 
       private

data/lib/resque/scheduler/server/views/delayed.erb

@@ -46,7 +46,7 @@
       <td><%= h(show_job_arguments(job['args'])) if job && delayed_timestamp_size == 1 %></td>
       <td>
         <% if job %>
-          <a href="<%=u URI("/delayed/jobs/#{job['class']}?args=" + URI.encode(job['args'].to_json)) %>">All schedules</a>
+          <a href="<%=u URI("/delayed/jobs/#{CGI.escape(job['class'])}?args=" + CGI.escape(job['args'].to_json)) %>">All schedules</a>
         <% end %>
       </td>
     </tr>

data/lib/resque/scheduler/server/views/search_form.erb

@@ -1,5 +1,5 @@
 <form method="POST" action="<%= u 'delayed/search' %>">
-  <input type='input' name='search' value="<%= params[:search] %>"/>
+  <input type='input' name='search' value="<%= h params[:search] %>"/>
   <input type='submit' value='Search'/>
 </form>
 

data/lib/resque/scheduler/server.rb

@@ -87,7 +87,7 @@ module Resque
         def delayed_jobs_klass
           begin
             klass = Resque::Scheduler::Util.constantize(params[:klass])
-            @args = JSON.load(URI.decode(params[:args]))
+            @args = JSON.load(CGI.unescape(params[:args]))
             @timestamps = Resque.scheduled_at(klass, *@args)
           rescue
             @timestamps = []

data/lib/resque/scheduler/version.rb

@@ -2,6 +2,6 @@
 
 module Resque
   module Scheduler
-    VERSION = '4.2.1'.freeze
+    VERSION = '4.5.0'.freeze
   end
 end

data/lib/resque/scheduler.rb

@@ -1,5 +1,6 @@
 # vim:fileencoding=utf-8
 
+require 'redis/errors'
 require 'rufus/scheduler'
 require_relative 'scheduler/configuration'
 require_relative 'scheduler/locking'
@@ -13,6 +14,9 @@ module Resque
     autoload :Extension, 'resque/scheduler/extension'
     autoload :Util, 'resque/scheduler/util'
     autoload :VERSION, 'resque/scheduler/version'
+    INTERMITTENT_ERRORS = [
+      Errno::EAGAIN, Errno::ECONNRESET, Redis::CannotConnectError, Redis::TimeoutError
+    ].freeze
 
     private
 
@@ -44,13 +48,7 @@ module Resque
         $stdout.sync = true
         $stderr.sync = true
 
-        # Load the schedule into rufus
-        # If dynamic is set, load that schedule otherwise use normal load
-        if dynamic
-          reload_schedule!
-        else
-          load_schedule!
-        end
+        was_master = nil
 
         begin
           @th = Thread.current
@@ -58,11 +56,21 @@ module Resque
           # Now start the scheduling part of the loop.
           loop do
             begin
-              if master?
+              # Check on changes to master/child
+              @am_master = master?
+              if am_master != was_master
+                procline am_master ? 'Master scheduler' : 'Child scheduler'
+
+                # Load schedule because changed
+                reload_schedule!
+              end
+
+              if am_master
                 handle_delayed_items
                 update_schedule if dynamic
               end
-            rescue Errno::EAGAIN, Errno::ECONNRESET, Redis::CannotConnectError => e
+              was_master = am_master
+            rescue *INTERMITTENT_ERRORS => e
               log! e.message
               release_master_lock
             end
@@ -99,7 +107,7 @@ module Resque
         Resque.schedule.each do |name, config|
           load_schedule_job(name, config)
         end
-        Resque.redis.del(:schedules_changed)
+        Resque.redis.del(:schedules_changed) if am_master && dynamic
         procline 'Schedules Loaded'
       end
 
@@ -141,11 +149,7 @@ module Resque
             args = [args, nil, job: true] if args.is_a?(::String)
 
             job = rufus_scheduler.send(interval_type, *args) do
-              if master?
-                log! "queueing #{config['class']} (#{name})"
-                Resque.last_enqueued_at(name, Time.now.to_s)
-                enqueue(config)
-              end
+              enqueue_recurring(name, config)
             end
             @scheduled_jobs[name] = job
             interval_defined = true
@@ -206,7 +210,7 @@ module Resque
         loop do
           handle_shutdown do
             # Continually check that it is still the master
-            item = enqueue_next_item(timestamp) if master?
+            item = enqueue_next_item(timestamp) if am_master
           end
           # continue processing until there are no more ready items in this
           # timestamp
@@ -371,7 +375,13 @@ module Resque
         true
       end
 
+      def stop_rufus_scheduler
+        rufus_scheduler.shutdown(:wait)
+        rufus_scheduler.join
+      end
+
       def before_shutdown
+        stop_rufus_scheduler
         release_master_lock
       end
 
@@ -417,6 +427,14 @@ module Resque
 
       private
 
+      def enqueue_recurring(name, config)
+        if am_master
+          log! "queueing #{config['class']} (#{name})"
+          enqueue(config)
+          Resque.last_enqueued_at(name, Time.now.to_s)
+        end
+      end
+
       def app_str
         app_name ? "[#{app_name}]" : ''
       end
@@ -432,6 +450,11 @@ module Resque
       def internal_name
         "resque-scheduler-#{Resque::Scheduler::VERSION}"
       end
+
+      def am_master
+        @am_master = master? unless defined?(@am_master)
+        @am_master
+      end
     end
   end
 end