resque-scheduler 5.1.0 → 5.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6aa6e1f1a8fe48f091032409157797e48011e473fca30c8fd12d18d05517de49
-  data.tar.gz: cbcc8d19930096797a1d028a9765b9c7c0b65ae023011ccec64cf8005420f183
+  metadata.gz: 7a49c095f83e23c0bb20e1dffe6038c64a50376d89a0123436eae4233a74e390
+  data.tar.gz: 80bd2fb3055d7904d0d62f47f3683c5c6a6f5c4d7236d7311b29ee7c6d2f1174
 SHA512:
-  metadata.gz: a3842dbcedbb2768e715aef2355f11333cadfa101ccb5a1475dbda155005d494408e0554c027ba5cf72afcabc7dcd3778125b9f512680850c875efbccb94f64f
-  data.tar.gz: 9623d1a81de5c0e1179e7be932ad6a19641dbcbc57c7b9034fc531160ae6e896374611d5d0f1dc64a710c27293cbc0e1db0eda9f6e5ad4c5ed8208398e20cdae
+  metadata.gz: c206ceb2414921369b7fd89800f95a8e3583226d709fd6d92effbfa69c1284711081cd46189b05aee1d14fdc3161f3d8fee0d7588c4e81a0a2cff6a791c0e415
+  data.tar.gz: f99d4d481f7ec3de91bef5a3b5ead66474e0f1df72a68bb705058de727ed448cd60cae39b901fd0b3edd4d02004fb87147e08a7b265517ca2b5109e194d7fb54

data/CHANGELOG.md

@@ -2,6 +2,12 @@
 
 **ATTN**: This project uses [semantic versioning](http://semver.org/).
 
+## [5.1.1] - 2026-11-02
+### Fixed
+* Enqueue batched delayed jobs outside the Redis transaction so a conflicted batch is no longer enqueued twice and enqueue hooks can use Redis by @neuromanskie in #850
+* Skip due delayed items when not master instead of spinning on them by @neuromanskie in #852
+* Load the web UI after Rails initialization to avoid a circular require warning by @neuromanskie in #854
+
 ## [5.1.0] - 2026-10-28
 ### Breaking Changes
 * Ruby requirement: `>= 3.0.0` → `>= 3.2.0` by @neuromanskie in #837

data/README.md

@@ -135,7 +135,7 @@ environment variables:
 * `BACKGROUND` - [Run in the background](#running-in-the-background) if
 non-empty (via `Process.daemon`, if supported) (default `false`)
 * `DELAYED_REQUEUE_BATCH_SIZE` - Set the delayed job batch size if enabled (default `100`). If `<= 1`, this disables batching.
-* `DISABLE_DELAYED_REQUEUE_BATCHES` - Disable batched delayed job queuing (default `false`) - [See section below on consequences](#batched-delayed-job-and-resque-enqueue-hooks)
+* `DISABLE_DELAYED_REQUEUE_BATCHES` - Disable batched delayed job queuing (default `false`)
 * `DYNAMIC_SCHEDULE` - Enables [dynamic scheduling](#dynamic-schedules)
 if non-empty (default `false`)
 * `RAILS_ENV` - Environment to use in procline (`$0`) (default empty)
@@ -784,35 +784,6 @@ This table explains the version requirements for rufus-scheduler
 | `~> 4.0`         | `~> 3.0`        |
 | `< 4.0`          | `~> 2.0`        |
 
-##### Batched delayed job and resque enqueue hooks
-
-Batching delayed job queuing can speed up when per-second job counts grows,
-avoiding situations that may cause delayed enqueues to fall behind. This
-batching wraps enqueues in a `multi` pipeline, making far fewer roundtrips to
-the server.
-
-However, in `redis` gem `>= 4.0`, any operations to Redis within the `multi`
-block must use the multi handle so that the actions are captured. Resque's hooks
-do not currently have a way to pass this around, and so compatibility with other
-Resque plugins or hooks which access Redis at enqueue time is impacted with
-batch mode (A main class of plugins this affects are single runner locks which
-use redis for their locking mechanism). In these cases, you should consider
-disabling the batching by setting the `DISABLE_DELAYED_REQUEUE_BATCHES`
-environment variable to `true`.
-
-Detecting when this occurs can be tricky, you must watch for logs
-emitted by your `resque-scheduler` process such as:
-
-```text
-Redis::CommandError: ERR
-MULTI calls can not be nested
-NoMethodError: undefined method nil? for <Redis::Future
-NoMethodError: undefined method `to_i' for <Redis::Future
-```
-
-If you're in this situation, you will also see delayed jobs to not be enqueued
-(they will get dropped). Disabling batching is the workaround.
-
 ### Contributing
 
 See [CONTRIBUTING.md](CONTRIBUTING.md)

data/lib/resque/scheduler/railtie.rb

@@ -1,8 +1,10 @@
-require 'resque/scheduler/server'
-
 module Resque
   module Scheduler
     class Railtie < Rails::Railtie
+      config.after_initialize do
+        require 'resque/scheduler/server'
+      end
+
       rake_tasks do
         require 'resque/scheduler/tasks'
       end

data/lib/resque/scheduler/version.rb

@@ -2,6 +2,6 @@
 
 module Resque
   module Scheduler
-    VERSION = '5.1.0'.freeze
+    VERSION = '5.1.1'.freeze
   end
 end

data/lib/resque/scheduler.rb

@@ -198,7 +198,7 @@ module Resque
       # at_time - Time to start scheduling items (default: now).
       def handle_delayed_items(at_time = nil)
         timestamp = Resque.next_delayed_timestamp(at_time)
-        if timestamp
+        if timestamp && am_master
           procline 'Processing Delayed Items'
           until timestamp.nil?
             enqueue_delayed_items_for_timestamp(timestamp)
@@ -235,11 +235,10 @@ module Resque
           actual_batch_size = 0
 
           handle_shutdown do
-            # Continually check that it is still the master
             if am_master
               if batch_delayed_items?
                 actual_batch_size = enqueue_items_in_batch_for_timestamp(timestamp, batch_size)
-                log "queued batch of #{actual_batch_size} jobs" if actual_batch_size != -1
+                log "queued batch of #{actual_batch_size} jobs"
               else
                 item = enqueue_next_item(timestamp)
                 actual_batch_size = item.nil? ? 0 : 1
@@ -251,8 +250,6 @@ module Resque
 
           # continue processing until there are no more items in this
           # timestamp. If we don't have a full batch, this is the last one.
-          # This also breaks us in the event of a redis transaction failure
-          # i.e. enqueue_items_in_batch_for_timestamp returned -1
           break if actual_batch_size < batch_size
         end
 
@@ -266,45 +263,32 @@ module Resque
       def enqueue_items_in_batch_for_timestamp(timestamp, batch_size)
         timestamp_bucket_key = timestamp_key(timestamp)
 
-        encoded_jobs_to_requeue = Resque.redis.lrange(timestamp_bucket_key, 0, batch_size - 1)
-
-        # Watch is used to ensure that the timestamp bucket we are operating on
-        # is not altered by any other clients between the watch call and when we call exec
-        # (to execute the multi block). We should error catch on the redis.exec return value
-        # as that will indicate if the entire transaction was aborted or not. Though we should
-        # be safe as our ltrim is inside the multi block and therefore also would have been
-        # aborted. So nothing would have been queued, but also nothing lost from the bucket.
-        watch_result = Resque.redis.watch(timestamp_bucket_key) do
-          Resque.redis.multi do |pipeline|
-            encoded_jobs_to_requeue.each do |encoded_job|
-              pipeline.srem("timestamps:#{encoded_job}", timestamp_bucket_key)
-
-              decoded_job = Resque.decode(encoded_job)
-              enqueue(decoded_job, :delayed)
-            end
+        return 0 unless (encoded_jobs = take_delayed_batch(timestamp_bucket_key, batch_size))
 
-            pipeline.ltrim(timestamp_bucket_key, batch_size, -1)
-          end
+        encoded_jobs.each do |encoded_job|
+          enqueue(Resque.decode(encoded_job), :delayed)
         end
 
-        # Did the multi block successfully remove from this timestamp and enqueue the jobs?
-        success = !watch_result.nil?
-
-        # If this was the last batch in this timestamp bucket, clean up
-        if success && encoded_jobs_to_requeue.count < batch_size
+        if encoded_jobs.count < batch_size
           Resque.clean_up_timestamp(timestamp_bucket_key, timestamp)
         end
 
-        unless success
-          # Our batched transaction failed in Redis due to the timestamp_bucket_key value
-          # being modified while we built our multi block. We return -1 to ensure we break
-          # out of the loop iterating on this timestamp so it can be re-processed via the
-          # loop in handle_delayed_items.
-          return -1
-        end
+        encoded_jobs.count
+      end
 
-        # will return 0 if none were left to batch
-        encoded_jobs_to_requeue.count
+      # Returns nil if the timestamp bucket changed before the batch was taken.
+      def take_delayed_batch(timestamp_bucket_key, batch_size)
+        Resque.redis.watch(timestamp_bucket_key) do
+          encoded_jobs = Resque.redis.lrange(timestamp_bucket_key, 0, batch_size - 1)
+          taken = Resque.redis.multi do |transaction|
+            encoded_jobs.each do |encoded_job|
+              transaction.srem("timestamps:#{encoded_job}", [timestamp_bucket_key])
+            end
+            transaction.ltrim(timestamp_bucket_key, batch_size, -1)
+          end
+
+          encoded_jobs if taken
+        end
       end
 
       # Enqueues a job like enqueue_from_config, but hands errors to the failure

data/resque-scheduler.gemspec

@@ -52,6 +52,7 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency 'ostruct'
   spec.add_development_dependency 'pry'
   spec.add_development_dependency 'rack-test'
+  spec.add_development_dependency 'railties'
   spec.add_development_dependency 'rake'
   spec.add_development_dependency 'simplecov'
   spec.add_development_dependency 'test-unit'

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: resque-scheduler
 version: !ruby/object:Gem::Version
-  version: 5.1.0
+  version: 5.1.1
 platform: ruby
 authors:
 - Ben VandenBos
@@ -140,6 +140,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: railties
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement