sidekiq-unique-jobs 7.0.0.beta29 → 7.0.4

data/README.md

@@ -3,9 +3,12 @@
 <!-- MarkdownTOC -->
 
 - [Introduction](#introduction)
-- [Requirements](#requirements)
-- [Installation](#installation)
+- [Usage](#usage)
+  - [Installation](#installation)
+  - [Add the middleware](#add-the-middleware)
+  - [Your first worker](#your-first-worker)
 - [Support Me](#support-me)
+- [Requirements](#requirements)
 - [General Information](#general-information)
 - [Global Configuration](#global-configuration)
   - [debug_lua](#debug_lua)
@@ -21,6 +24,8 @@
   - [lock_prefix](#lock_prefix)
   - [lock_info](#lock_info)
 - [Worker Configuration](#worker-configuration)
+  - [lock_info](#lock_info-1)
+  - [lock_prefix](#lock_prefix-1)
   - [lock_ttl](#lock_ttl-1)
   - [lock_timeout](#lock_timeout-1)
   - [unique_across_queues](#unique_across_queues)
@@ -39,13 +44,15 @@
   - [replace](#replace)
   - [Reschedule](#reschedule)
   - [Custom Strategies](#custom-strategies)
-- [Usage](#usage)
+- [Usage](#usage-1)
   - [Finer Control over Uniqueness](#finer-control-over-uniqueness)
   - [After Unlock Callback](#after-unlock-callback)
   - [Logging](#logging)
   - [Cleanup Dead Locks](#cleanup-dead-locks)
   - [Other Sidekiq gems](#other-sidekiq-gems)
+    - [apartment-sidekiq](#apartment-sidekiq)
     - [sidekiq-global_id](#sidekiq-global_id)
+    - [sidekiq-status](#sidekiq-status)
 - [Debugging](#debugging)
   - [Sidekiq Web](#sidekiq-web)
     - [Show Locks](#show-locks)
@@ -61,7 +68,7 @@
 
 ## Introduction
 
-This gem adds unique constraints to the sidekiq queues. The uniqueness is achieved by acquiring locks for a hash of a queue name, a worker class, and job's arguments. Only one lock for a given hash can be acquired. What happens when a lock can't be acquired is governed by a chosen strategy.
+This gem adds unique constraints to the sidekiq queues. The uniqueness is achieved by acquiring locks for a hash of a queue name, a worker class, and job's arguments. By default, only one lock for a given hash can be acquired. What happens when a lock can't be acquired is governed by a chosen `on_conflict`strategy.
 
 This is the documentation for the master branch. You can find the documentation for each release by navigating to its tag.
 
@@ -71,20 +78,9 @@ Here are links to some of the old versions
 - [v5.0.10](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v5.0.10)
 - [v4.0.18](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v4.0.18)
 
-## Requirements
-
-- Sidekiq `>= 4.0` (`>= 5.2` recommended)
-- Ruby:
-  - MRI `>= 2.3` (`>= 2.5` recommended)
-  - JRuby `>= 9.0` (`>= 9.2` recommended)
-  - Truffleruby
-- Redis Server `>= 3.0.2` (`>= 3.2` recommended)
-- [ActiveJob officially not supported][48]
-- [redis-namespace officially not supported][49]
-
-See [Sidekiq requirements][24] for detailed requirements of Sidekiq itself (be sure to check the right sidekiq version).
+## Usage
 
-## Installation
+### Installation
 
 Add this line to your application's Gemfile:
 
@@ -98,16 +94,83 @@ And then execute:
 bundle
 ```
 
-Or install it yourself as:
+### Add the middleware
 
-```bash
-gem install sidekiq-unique-jobs
+Before v7, the middleware was configured automatically. Since some people reported issues with other gems (see [Other Sidekiq Gems](#other-sidekiq-gems)) it was decided to give full control over to the user.
+
+*NOTE* if you want to use the reaper you also need to configure the server middleware.
+
+[A full example](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/myapp/config/initializers/sidekiq.rb#L12)
+
+```ruby
+Sidekiq.configure_server do |config|
+  config.redis = { url: ENV["REDIS_URL"], driver: :hiredis }
+
+  config.client_middleware do |chain|
+    chain.add SidekiqUniqueJobs::Middleware::Client
+  end
+
+  config.server_middleware do |chain|
+    chain.add SidekiqUniqueJobs::Middleware::Server
+  end
+
+  SidekiqUniqueJobs::Server.configure(config)
+end
+
+Sidekiq.configure_client do |config|
+  config.redis = { url: ENV["REDIS_URL"], driver: :hiredis }
+
+  config.client_middleware do |chain|
+    chain.add SidekiqUniqueJobs::Middleware::Client
+  end
+end
 ```
 
+### Your first worker
+
+```ruby
+# frozen_string_literal: true
+
+class UntilExecutedWorker
+  include Sidekiq::Worker
+
+  sidekiq_options queue: :special,
+                  retry: false,
+                  lock: :until_executed,
+                  lock_info: true,
+                  lock_timeout: 0,
+                  lock_prefix: "special",
+                  lock_ttl: 0,
+                  lock_limit: 5
+
+  def perform
+    logger.info("cowboy")
+    sleep(1) # hardcore processing
+    logger.info("beebop")
+  end
+end
+
+```
+
+You can read more about the worker configuration in [Worker Configuration](#worker-configuration) below.
+
 ## Support Me
 
 Want to show me some ❤️ for the hard work I do on this gem? You can use the following PayPal link: [https://paypal.me/mhenrixon1](https://paypal.me/mhenrixon1). Any amount is welcome and let me tell you it feels good to be appreciated. Even a dollar makes me super excited about all of this.
 
+## Requirements
+
+- Sidekiq `>= 5.0` (`>= 5.2` recommended)
+- Ruby:
+  - MRI `>= 2.5` (`>= 2.6` recommended)
+  - JRuby `>= 9.0` (`>= 9.2` recommended)
+  - Truffleruby
+- Redis Server `>= 3.2` (`>= 5.0` recommended)
+- [ActiveJob officially not supported][48]
+- [redis-namespace officially not supported][49]
+
+See [Sidekiq requirements][24] for detailed requirements of Sidekiq itself (be sure to check the right sidekiq version).
+
 ## General Information
 
 See [Interaction w/ Sidekiq](https://github.com/mhenrixon/sidekiq-unique-jobs/wiki/How-this-gem-interacts-with-Sidekiq) on how the gem interacts with Sidekiq.
@@ -122,16 +185,16 @@ Configure SidekiqUniqueJobs in an initializer or the sidekiq initializer on appl
 
 ```ruby
 SidekiqUniqueJobs.configure do |config|
-  config.debug_lua       = true
-  config.lock_info       = true
-  config.lock_ttl        = 10.minutes
-  config.lock_timeout    = 10.minutes
-  config.logger          = Sidekiq.logger
-  config.max_history     = 10_000
-  config.reaper          = :lua
-  config.reaper_count    = 100
-  config.reaper_interval = 10
-  config.reaper_timeout  = 5
+  config.logger = Sidekiq.logger # default, change at your own discretion
+  config.debug_lua       = false # Turn on when debugging
+  config.lock_info       = false # Turn on when debugging
+  config.lock_ttl        = 600   # Expire locks after 10 minutes
+  config.lock_timeout    = nil   # turn off lock timeout
+  config.max_history     = 0     # Turn on when debugging
+  config.reaper          = :ruby # :ruby, :lua or :none/nil
+  config.reaper_count    = 1000  # Stop reaping after this many keys
+  config.reaper_interval = 600   # Reap orphans every 10 minutes
+  config.reaper_timeout  = 150   # Timeout reaper after 2.5 minutes
 end
 ```
 
@@ -253,11 +316,27 @@ Using lock info will create an additional key for the lock with a json object co
 
 ## Worker Configuration
 
+### lock_info
+
+Lock info gathers information about a specific lock. It collects things like which `lock_args` where used to compute the `lock_digest` that is used for maintaining uniqueness.
+
+```ruby
+sidekiq_options lock_info: false # this is the default, set to true to turn on
+```
+
+### lock_prefix
+
+Use if you want a different key prefix for the keys in redis.
+
+```ruby
+sidekiq_options lock_prefix: "uniquejobs" # this is the default value
+```
+
 ### lock_ttl
 
 Lock TTL decides how long to wait after the job has been successfully processed before making it possible to reuse that lock.
 
-Since `v6.0.11` the other locks will expire after the server is done processing.
+Starting from `v7` the expiration will take place when the job is pushed to the queue.
 
 ```ruby
 sidekiq_options lock_ttl: nil # default - don't expire keys
@@ -278,6 +357,8 @@ sidekiq_options lock_timeout: nil # lock indefinitely, this process won't contin
 
 This configuration option is slightly misleading. It doesn't disregard the queue on other jobs. Just on itself, this means that a worker that might schedule jobs into multiple queues will be able to have uniqueness enforced on all queues it is pushed to.
 
+This is mainly intended for `Worker.set(queue: :another).perform_async`.
+
 ```ruby
 class Worker
   include Sidekiq::Worker
@@ -428,7 +509,7 @@ The last one is log which can be be used with the lock `UntilExecuted` and `Unti
 It is possible for locks to have different conflict strategy for the client and server. This is useful for `:until_and_while_executing`.
 
 ```ruby
-sidekiq_options lock: :until_and_while_executing, 
+sidekiq_options lock: :until_and_while_executing,
                 on_conflict: { client: :log, server: :reject }
 ```
 
@@ -628,7 +709,7 @@ class MyWorker
 end
 ```
 
-Starting in v5.1, Sidekiq can also fire a global callback when a job dies:
+Starting in v5.1, Sidekiq can also fire a global callback when a job dies: In version 7, this is handled automatically for you. You don't need to add a death handler, if you configure v7 like in [Add the middleware](#add-the-middleware) you don't have to worry about the below.
 
 ```ruby
 Sidekiq.configure_server do |config|
@@ -641,10 +722,30 @@ end
 
 ### Other Sidekiq gems
 
+#### apartment-sidekiq
+
+It was reported in [#536](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/536) that the order of the Sidekiq middleware needs to be as follows.
+
+```ruby
+Sidekiq.client_middleware do |chain|
+  chain.add Apartment::Sidekiq::Middleware::Client
+  chain.add SidekiqUniqueJobs::Middleware::Client
+end
+
+Sidekiq.server_middleware do |chain|
+  chain.add Apartment::Sidekiq::Middleware::Server
+  chain.add SidekiqUniqueJobs::Middleware::Server
+end
+```
+
+The reason being that this gem needs to be configured AFTER the apartment gem or the apartment will not be able to be considered for uniqueness
+
 #### sidekiq-global_id
 
 It was reported in [#235](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/235) that the order of the Sidekiq middleware needs to be as follows.
 
+For a working setup check the following [file](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/myapp/config/sidekiq.rb#L12).
+
 ```ruby
 Sidekiq.client_middleware do |chain|
   chain.add Sidekiq::GlobalId::ClientMiddleware
@@ -652,12 +753,44 @@ Sidekiq.client_middleware do |chain|
 end
 
 Sidekiq.server_middleware do |chain|
-  chain.add SidekiqUniqueJobs::Middleware::Server
   chain.add Sidekiq::GlobalId::ServerMiddleware
+  chain.add SidekiqUniqueJobs::Middleware::Server
+end
+```
+
+The reason for this is that the global id needs to be set before the unique jobs middleware runs. Otherwise that won't be available for uniqueness.
+
+#### sidekiq-status
+
+It was reported in [#564](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/564) that the order of the middleware needs to be as follows.
+
+```ruby
+# Thanks to @ArturT for the correction
+
+Sidekiq.configure_server do |config|
+  config.client_middleware do |chain|
+    chain.add SidekiqUniqueJobs::Middleware::Client
+    chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
+  end
+
+  config.server_middleware do |chain|
+    chain.add Sidekiq::Status::ServerMiddleware, expiration: 30.minutes
+    chain.add SidekiqUniqueJobs::Middleware::Server
+  end
+
+  SidekiqUniqueJobs::Server.configure(config)
+end
+
+
+Sidekiq.configure_client do |config|
+  config.client_middleware do |chain|
+    chain.add SidekiqUniqueJobs::Middleware::Client
+    chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
+  end
 end
 ```
 
-For a working setup check the following [file](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/945c4c4c517168d49e3f8ee952fcc9c430865635/myapp/config/initializers/sidekiq.rb#L8)
+The reason for this is that if a job is duplicated it shouldn't end up with the status middleware at all. Status is just a monitor so to prevent clashes, leftovers and ensure cleanup. The status middleware should run after uniqueness on client and before on server. This will lead to less surprises.
 
 ## Debugging
 
@@ -728,7 +861,6 @@ If you are not using RSpec (a lot of people prefer minitest or test unit) you ca
 assert SidekiqUniqueJobs.validate_worker!(BadWorker.get_sidekiq_options)
 ```
 
-
 ### Uniqueness
 
 This has been probably the most confusing part of this gem. People get really confused with how unreliable the unique jobs have been. I there for decided to do what Mike is doing for sidekiq enterprise. Read the section about unique jobs: [Enterprise unique jobs][]
@@ -771,7 +903,7 @@ RSpec.describe Workers::CoolOne do
 end
 ```
 
-It is recommened to leave the uniqueness testing to the gem maintainers. If you care about how the gem is integration tested have a look at the following specs:
+It is recommended to leave the uniqueness testing to the gem maintainers. If you care about how the gem is integration tested have a look at the following specs:
 
 - [spec/integration/sidekiq_unique_jobs/lock/until_and_while_executing_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/integration/sidekiq_unique_jobs/lock/until_and_while_executing_spec.rb)
 - [spec/integration/sidekiq_unique_jobs/lock/until_executed_spec.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/integration/sidekiq_unique_jobs/lock/until_executed_spec.rb)

data/lib/sidekiq-unique-jobs.rb

@@ -1,5 +1,3 @@
 # frozen_string_literal: true
 
 require "sidekiq_unique_jobs"
-
-SidekiqUniqueJobs::Middleware.configure

data/lib/sidekiq_unique_jobs.rb

@@ -3,9 +3,10 @@
 require "brpoplpush/redis_script"
 require "concurrent/future"
 require "concurrent/promises"
-require "concurrent/timer_task"
 require "concurrent/map"
 require "concurrent/mutable_struct"
+require "concurrent/timer_task"
+require "concurrent/executor/ruby_single_thread_executor"
 require "digest"
 require "digest/sha1"
 require "erb"
@@ -14,6 +15,7 @@ require "json"
 require "pathname"
 require "sidekiq"
 
+require "sidekiq_unique_jobs/timer_task"
 require "sidekiq_unique_jobs/version"
 require "sidekiq_unique_jobs/version_check"
 require "sidekiq_unique_jobs/constants"
@@ -71,3 +73,4 @@ require "sidekiq_unique_jobs/config"
 require "sidekiq_unique_jobs/sidekiq_unique_jobs"
 require "sidekiq_unique_jobs/update_version"
 require "sidekiq_unique_jobs/upgrade_locks"
+require "sidekiq_unique_jobs/server"

data/lib/sidekiq_unique_jobs/batch_delete.rb

@@ -44,7 +44,7 @@ module SidekiqUniqueJobs
     #
     # @return [void]
     #
-    def self.call(digests, conn)
+    def self.call(digests, conn = nil)
       new(digests, conn).call
     end
 

data/lib/sidekiq_unique_jobs/changelog.rb

@@ -7,6 +7,13 @@ module SidekiqUniqueJobs
   # @author Mikael Henriksson <mikael@mhenrixon.com>
   #
   class Changelog < Redis::SortedSet
+    #
+    # @return [Integer] the number of matches to return by default
+    DEFAULT_COUNT = 1_000
+    #
+    # @return [String] the default pattern to use for matching
+    SCAN_PATTERN  = "*"
+
     def initialize
       super(CHANGELOGS)
     end
@@ -34,10 +41,10 @@ module SidekiqUniqueJobs
     #
     # @return [Array<Hash>] an array of entries
     #
-    def entries(pattern: "*", count: nil)
+    def entries(pattern: SCAN_PATTERN, count: DEFAULT_COUNT)
       options = {}
-      options[:match] = pattern if pattern
-      options[:count] = count if count
+      options[:match] = pattern
+      options[:count] = count
 
       redis do |conn|
         conn.zscan_each(key, **options).to_a.map { |entry| load_json(entry[0]) }
@@ -53,7 +60,7 @@ module SidekiqUniqueJobs
     #
     # @return [Array<Integer, Integer, Array<Hash>] the total size, next cursor and changelog entries
     #
-    def page(cursor, pattern: "*", page_size: 100)
+    def page(cursor: 0, pattern: "*", page_size: 100)
       redis do |conn|
         total_size, result = conn.multi do
           conn.zcard(key)

data/lib/sidekiq_unique_jobs/constants.rb

@@ -11,6 +11,7 @@ module SidekiqUniqueJobs
   AT                    ||= "at"
   CHANGELOGS            ||= "uniquejobs:changelog"
   CLASS                 ||= "class"
+  CREATED_AT            ||= "created_at"
   DEAD_VERSION          ||= "uniquejobs:dead"
   DIGESTS               ||= "uniquejobs:digests"
   ERRORS                ||= "errors"

data/lib/sidekiq_unique_jobs/digests.rb

@@ -78,7 +78,7 @@ module SidekiqUniqueJobs
     def entries(pattern: SCAN_PATTERN, count: DEFAULT_COUNT)
       options = {}
       options[:match] = pattern
-      options[:count] = count if count
+      options[:count] = count
 
       result = redis { |conn| conn.zscan_each(key, **options).to_a }
 

data/lib/sidekiq_unique_jobs/json.rb

@@ -15,11 +15,17 @@ module SidekiqUniqueJobs
     # @return [Object]
     #
     def load_json(string)
-      return unless string && !string.empty?
+      return if string.nil? || string.empty?
 
       ::JSON.parse(string)
     end
 
+    def safe_load_json(string)
+      return string if string.is_a?(Hash)
+
+      load_json(string)
+    end
+
     #
     # Dumps an object into a JSON string
     #

data/lib/sidekiq_unique_jobs/lock.rb

@@ -53,7 +53,7 @@ module SidekiqUniqueJobs
     #
     # Locks a job_id
     #
-    # @note intended only for testing purposez
+    # @note intended only for testing purposes
     #
     # @param [String] job_id a sidekiq JID
     # @param [Hash] lock_info information about the lock
@@ -73,6 +73,36 @@ module SidekiqUniqueJobs
       end
     end
 
+    #
+    # Create the :QUEUED key
+    #
+    # @note intended only for testing purposes
+    #
+    # @param [String] job_id a sidekiq JID
+    #
+    # @return [void]
+    #
+    def queue(job_id)
+      redis do |conn|
+        conn.lpush(key.queued, job_id)
+      end
+    end
+
+    #
+    # Create the :PRIMED key
+    #
+    # @note intended only for testing purposes
+    #
+    # @param [String] job_id a sidekiq JID
+    #
+    # @return [void]
+    #
+    def prime(job_id)
+      redis do |conn|
+        conn.lpush(key.primed, job_id)
+      end
+    end
+
     #
     # Unlock a specific job_id
     #