sidekiq-unique-jobs 6.0.0.rc3 → 6.0.0.rc4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 040dccf5a4139ee93bdc4d8b76d641b54fd9048ed4e82efe653e08ed4c5cadae
-  data.tar.gz: 64059716024f596926a052d05da8cd27ec10e5cde5186cf84c6db2dd3e6b039f
+  metadata.gz: 17113a3ab2a97217143984d5e401054568eae54dae1bb1e411f5c6c82d8c15c2
+  data.tar.gz: 4a75c98d2174368796e734c11eca12e0f54fbe7fb39c6d55affd2193026acb89
 SHA512:
-  metadata.gz: 84fc218ebc8d22ebcb1b94a01344b313c8bf1dce2f7d89961edd8dee2307143fa9c75fe09b5c0ea46b0f736e93dc603b683202172727d55873b2aa2944225cf6
-  data.tar.gz: e0f78dea666c477a2b64929f82e85edfe5dfb6e1b21a030b2890761a3225e1baa619fd15df04cb1bad4dbfca77ea3334f3d7784f54e1da56662b35f943b3c656
+  metadata.gz: 2655eebc021bb239a407739369f18bfe5df559c7182cdd104ce148c4678f5788c24a609d0b9187d2fa6b770408d6002ac5b61460d129f6df5d07dfb96496058c
+  data.tar.gz: 697a789d9e8499cdf2cb74cc8d27d39b9efe36e414eb605f047591fe87710f738ef5d889bdb3ba59c3fd9dcd13261a9463f38c564d32d54e681e596876a904f3

data/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,31 @@
+---
+name: Bug report
+about: Create a report to help us improve
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Current behavior**
+What happens instead of the expected behavior?
+
+**Worker class**
+
+```ruby
+class MyWorker
+  include Sidekiq::Worker
+  sidekiq_options unique: :until_executed, queue: :undefault
+  def perform(args); end
+
+  def self.unique_args(args)
+    # the way you consider unique arguments
+  end
+end
+```
+
+**Additional context**
+Add any other context about the problem here.

data/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,17 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

data/.gitignore

@@ -20,3 +20,7 @@ rails_example/spec/examples.txt
 .rspec_status
 
 /gemfiles/.bundle/
+
+/.byebug_history
+
+/.yardoc/

data/README.md

@@ -2,27 +2,20 @@
 
 The missing unique jobs for sidekiq
 
-# Documentation
+## Documentation
 
-This is the documentation for the master branch. You can find the documentation for each release by navigating to its tag: https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v5.0.10. 
+This is the documentation for the master branch. You can find the documentation for each release by navigating to its tag: https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v5.0.10.
 
 Below are links to the latest major versions (4 & 5):
+
 - [v5.0.10](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v5.0.10)
 - [v4.0.18](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v4.0.18)
 
 ## Requirements
 
-See https://github.com/mperham/sidekiq#requirements for what is required. Starting from 5.0.0 only sidekiq >= 4 is supported and support for MRI <=  2.1 is dropped. ActiveJob is not supported 
-
-Version 5 requires redis >= 3
-
-### ActiveJob
+See https://github.com/mperham/sidekiq#requirements for what is required. Starting from 5.0.0 only sidekiq >= 4 is supported and support for MRI <= 2.1 is dropped. ActiveJob is not supported
 
-Due to the simplicity of ActiveJob and the complexity of this game there is no officially supported way of doing ActiveJob. If you want to use uniqueness you should be using sidekiq directly. I know some projects started by using ActiveJob out of ignorance and someone has to do a whole lot of work to migrate the workers to use sidekiq directly... 
-
-If you are in this position and you can't figure it out; I have done such migrations for really big clients before. I am a consultant with a ton of experience on such jobs. My rate is fair and I am easy to get along with.
-
-If that is not an option I apologize. This gem won't support ActiveJob moving forward. It would require monkey patching too much.
+Version 6 requires Redis >= 3 and pure Sidekiq, no ActiveJob supported anymore. See [About ActiveJob](https://github.com/mhenrixon/sidekiq-unique-jobs/wiki/About-ActiveJob) for why.
 
 ## Installation
 
@@ -38,15 +31,17 @@ Or install it yourself as:
 
     $ gem install sidekiq-unique-jobs
 
-## Locking
+## General Information
+
+See [Interaction w/ Sidekiq](https://github.com/mhenrixon/sidekiq-unique-jobs/wiki/How-this-gem-interacts-with-Sidekiq) on how the gem interacts with Sidekiq.
 
-Sidekiq consists of a client and a server. The client is responsible for pushing jobs to the queue and the server is responsible for actually processing the jobs. When the client puts the job to the queue the middleware checks for uniqueness and creates a lock. When the server then processes the job that lock is released.
+See [Locking & Unlocking](https://github.com/mhenrixon/sidekiq-unique-jobs/wiki/Locking-&-Unlocking) for an overview of the differences on when the various lock types are locked and unlocked.
 
 ### Options
 
 #### Lock Expiration
 
-This is probably not the configuration option you want... 
+This is probably not the configuration option you want...
 
 Since the client and the server are disconnected and not running inside the same process, setting a lock expiration is probably not what you want. Any keys that are used by this gem WILL be removed at the time of the expiration. For jobs that are scheduled in the future the key will expire when that job is scheduled + whatever expiration you have set.
 
@@ -67,7 +62,54 @@ sidekiq_options lock_timeout: 5 # wait 5 seconds
 sidekiq_options lock_timeout: nil # lock indefinitely, this process won't continue until it gets a lock. VERY DANGEROUS!!
 ```
 
-#### 
+#### Unique Across Queues
+
+This configuration option is slightly misleading. It doesn't disregard the queue on other jobs. Just on itself, this means that a worker that might schedule jobs into multiple queues will be able to have uniqueness enforced on all queues it is pushed to.
+
+```ruby
+class Worker
+  include Sidekiq::Worker
+  
+  sidekiq_options: unique_across_queues: true, queue: 'default'
+
+  def perform(args); end
+end
+```
+
+Now if you push override the queue with `Worker.set(queue: 'another').perform_async(1)` it will still be considered unique when compared to `Worker.perform_async(1)` (that was actually pushed to the queue `default`).
+
+#### Unique Across Workers
+
+This configuration option is slightly misleading. It doesn't disregard the worker class on other jobs. Just on itself, this means that a worker that the worker class won't be used for generating the unique digest. The only way this option really makes sense is when you want to have uniqueness between two different worker classes.
+
+```ruby
+class WorkerOne
+  include Sidekiq::Worker
+  
+  sidekiq_options: unique_across_workers: true, queue: 'default'
+
+  def perform(args); end
+end
+
+class WorkerTwo
+  include Sidekiq::Worker
+  
+  sidekiq_options: unique_across_workers: true, queue: 'default'
+
+  def perform(args); end
+end
+
+
+WorkerOne.perform_async(1) 
+# => 'the jobs unique id'
+
+WorkerTwo.perform_async(1) 
+# => nil because WorkerOne just stole the lock
+```
+
+### Locks
+
+####
 
 ### Until Executing
 
@@ -83,14 +125,13 @@ sidekiq_options unique: :until_executing
 
 Locks from when the client pushes the job to the queue. Will be unlocked when the server has successfully processed the job.
 
-
 ```ruby
 sidekiq_options unique: :until_executed
 ```
 
 ### Until Timeout
 
-Locks from when the client pushes the job to the queue. Will be unlocked when the specified timeout has been reached. 
+Locks from when the client pushes the job to the queue. Will be unlocked when the specified timeout has been reached.
 
 ```ruby
 sidekiq_options unique: :until_expired
@@ -106,11 +147,9 @@ sidekiq_options unique: :until_and_while_executing
 
 ### While Executing
 
-With this lock type it is possible to put any number of these jobs on the queue, but as the server pops the job from the queue it will create a lock and then wait until other locks are done processing. It *looks* like multiple jobs are running at the same time but in fact the second job will only be waiting for the first job to finish.
+With this lock type it is possible to put any number of these jobs on the queue, but as the server pops the job from the queue it will create a lock and then wait until other locks are done processing. It _looks_ like multiple jobs are running at the same time but in fact the second job will only be waiting for the first job to finish.
 
-#### NOTE:
-
-Unless this job is configured with a `lock_timeout: nil` or `lock_timeout: > 0` then all jobs that are attempted to be executed will just be dropped without waiting.
+**NOTE** Unless this job is configured with a `lock_timeout: nil` or `lock_timeout: > 0` then all jobs that are attempted to be executed will just be dropped without waiting.
 
 ```ruby
 sidekiq_options unique: :while_executing, lock_timeout: nil
@@ -135,17 +174,9 @@ In the console you should see something like:
 10:33:04 worker.1 | 2017-04-23T08:33:04.973Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f INFO: done: 40.014 sec
 ```
 
-
-### Uniqueness Scope
-
-- Queue specific locks
-- Across all queues - [examples/unique_on_all_queues_job.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/examples/unique_on_all_queues_job.rb)
-- Across all workers - [examples/unique_across_workers_job.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/examples/unique_across_workers_job.rb)
-- Timed / Scheduled jobs
-
 ## Usage
 
-All that is required is that you specifically set the sidekiq option for *unique* to a valid value like below:
+All that is required is that you specifically set the sidekiq option for _unique_ to a valid value like below:
 
 ```ruby
 sidekiq_options unique: :while_executing
@@ -165,7 +196,7 @@ The method or the proc can return a modified version of args without the transie
 class UniqueJobWithFilterMethod
   include Sidekiq::Worker
   sidekiq_options unique: :until_and_while_executing,
-                  unique_args: :unique_args
+                  unique_args: :unique_args # this is default and will be used if such a method is defined
 
   def self.unique_args(args)
     [ args[0], args[2][:type] ]
@@ -185,9 +216,6 @@ class UniqueJobWithFilterProc
 end
 ```
 
-The previous problems with unique args being string in server and symbol in client is no longer a problem because the `UniqueArgs` class accounts for this and converts everything to json now. If you find an edge case please provide and example so that we can add coverage and fix it.
-
-
 It is also quite possible to ensure different types of unique args based on context. I can't vouch for the below example but see [#203](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/203) for the discussion.
 
 ```ruby
@@ -207,10 +235,12 @@ class UniqueJobWithFilterMethod
 end
 ```
 
-
 ### After Unlock Callback
 
-If you are using :after_yield as your unlock ordering, Unique Job offers a callback to perform some work after the block is yielded.
+If you need to perform any additional work after the lock has been released you can provide an `#after_unlock` instance method. The method will be called when the lock has been unlocked. Most times this means after yield but there are two exceptions to that. 
+
+**Exception 1:** UntilExecuting unlocks and calls back before yielding.
+**Exception 2:** UntilExpired expires eventually, no after_unlock hook is called.
 
 ```ruby
 class UniqueJobWithFilterMethod
@@ -222,12 +252,11 @@ class UniqueJobWithFilterMethod
   end
   ...
 end.
-
 ```
 
 ### Logging
 
-To see logging in sidekiq when duplicate payload has been filtered out you can enable on a per worker basis using the sidekiq options.  The default value is false
+To see logging in sidekiq when duplicate payload has been filtered out you can enable on a per worker basis using the sidekiq options. The default value is false
 
 ```ruby
 class UniqueJobWithFilterMethod
@@ -241,15 +270,19 @@ end
 ```
 
 ## Debugging
+
 There are two ways to display and remove keys regarding uniqueness. The console way and the command line way.
 
 ### Console
+
 Start the console with the following command `bundle exec jobs console`.
 
 #### List Unique Keys
+
 `keys '*', 100`
 
 #### Remove Unique Keys
+
 `del '*', 100, false` the dry_run and count parameters are both required. This is to have some type of protection against clearing out all uniqueness.
 
 ### Command Line

data/lib/sidekiq_unique_jobs/constants.rb

@@ -9,10 +9,11 @@ module SidekiqUniqueJobs
   LOCK_TIMEOUT_KEY          ||= 'lock_timeout'
   LOG_DUPLICATE_KEY         ||= 'log_duplicate_payload'
   QUEUE_KEY                 ||= 'queue'
+  UNIQUE_ACROSS_QUEUES_KEY  ||= 'unique_across_queues'
   UNIQUE_ACROSS_WORKERS_KEY ||= 'unique_across_workers'
   UNIQUE_ARGS_KEY           ||= 'unique_args'
   UNIQUE_DIGEST_KEY         ||= 'unique_digest'
   UNIQUE_KEY                ||= 'unique'
-  UNIQUE_ON_ALL_QUEUES_KEY  ||= 'unique_on_all_queues'
+  UNIQUE_ON_ALL_QUEUES_KEY  ||= 'unique_on_all_queues' # TODO: Remove in v6.1
   UNIQUE_PREFIX_KEY         ||= 'unique_prefix'
 end

data/lib/sidekiq_unique_jobs/lock/base_lock.rb

@@ -5,16 +5,18 @@ module SidekiqUniqueJobs
     class BaseLock
       include SidekiqUniqueJobs::Logging
 
-      def initialize(item, redis_pool = nil)
+      def initialize(item, callback, redis_pool = nil)
         @item       = prepare_item(item)
+        @callback   = callback
         @redis_pool = redis_pool
+        @operative  = true
       end
 
       def lock
         locksmith.lock(item[LOCK_TIMEOUT_KEY])
       end
 
-      def execute(_callback = nil)
+      def execute
         raise NotImplementedError, "##{__method__} needs to be implemented in #{self.class}"
       end
 
@@ -36,20 +38,19 @@ module SidekiqUniqueJobs
 
       private
 
-      attr_reader :item, :redis_pool, :operative
+      attr_reader :item, :redis_pool, :operative, :callback
 
       def locksmith
         @locksmith ||= SidekiqUniqueJobs::Locksmith.new(item, redis_pool)
       end
 
-      def using_protection(callback)
-        @operative = true
+      def with_cleanup
         yield
       rescue Sidekiq::Shutdown
         @operative = false
         raise
       ensure
-        unlock_and_callback(callback)
+        unlock_with_callback
       end
 
       def prepare_item(item)
@@ -60,23 +61,23 @@ module SidekiqUniqueJobs
         item
       end
 
-      def unlock_and_callback(callback)
-        return notify_about_manual_unlock unless operative
-        unlock
-        delete
-
-        return notify_about_manual_unlock if locked?
-        callback_safely(callback)
-      end
-
       def notify_about_manual_unlock
         log_fatal("the unique_key: #{item[UNIQUE_DIGEST_KEY]} needs to be unlocked manually")
+        false
+      end
+
+      def unlock_with_callback
+        return notify_about_manual_unlock unless operative
+        return notify_about_manual_unlock unless unlock
+
+        callback_safely
+        item[JID_KEY]
       end
 
-      def callback_safely(callback)
-        callback.call
+      def callback_safely
+        callback&.call
       rescue StandardError
-        log_warn("the callback for unique_key: #{item[UNIQUE_DIGEST_KEY]} failed!")
+        log_warn("The lock for #{item[UNIQUE_DIGEST_KEY]} has been released but the #after_unlock callback failed!")
         raise
       end
     end

data/lib/sidekiq_unique_jobs/lock/until_and_while_executing.rb

@@ -3,17 +3,17 @@
 module SidekiqUniqueJobs
   class Lock
     class UntilAndWhileExecuting < BaseLock
-      def execute(callback)
+      def execute
         return unless locked?
         unlock
 
-        runtime_lock.execute(callback) do
+        runtime_lock.execute do
           yield if block_given?
         end
       end
 
       def runtime_lock
-        @runtime_lock ||= SidekiqUniqueJobs::Lock::WhileExecuting.new(item, redis_pool)
+        @runtime_lock ||= SidekiqUniqueJobs::Lock::WhileExecuting.new(item, callback, redis_pool)
       end
     end
   end

data/lib/sidekiq_unique_jobs/lock/until_executed.rb

@@ -5,12 +5,9 @@ module SidekiqUniqueJobs
     class UntilExecuted < BaseLock
       OK ||= 'OK'
 
-      def execute(callback)
+      def execute
         return unless locked?
-        using_protection(callback) do
-          yield if block_given?
-        end
-        unlock
+        with_cleanup { yield if block_given? }
       end
     end
   end

data/lib/sidekiq_unique_jobs/lock/until_executing.rb

@@ -3,9 +3,8 @@
 module SidekiqUniqueJobs
   class Lock
     class UntilExecuting < BaseLock
-      def execute(callback)
-        unlock
-        callback.call
+      def execute
+        unlock_with_callback
         yield if block_given?
       end
     end

data/lib/sidekiq_unique_jobs/lock/until_expired.rb

@@ -7,10 +7,10 @@ module SidekiqUniqueJobs
         true
       end
 
-      def execute(callback)
+      def execute
         return unless locked?
         yield if block_given?
-        callback.call
+        # this lock does not handle after_unlock since we don't know when that would
       end
     end
   end

data/lib/sidekiq_unique_jobs/lock/while_executing.rb

@@ -5,8 +5,8 @@ module SidekiqUniqueJobs
     class WhileExecuting < BaseLock
       RUN_SUFFIX ||= ':RUN'
 
-      def initialize(item, redis_pool = nil)
-        super(item, redis_pool)
+      def initialize(item, callback, redis_pool = nil)
+        super(item, callback, redis_pool)
         append_unique_key_suffix
       end
 
@@ -17,12 +17,9 @@ module SidekiqUniqueJobs
       end
 
       # Locks the job with the RUN_SUFFIX appended
-      def execute(callback)
-        locksmith.lock(item[LOCK_TIMEOUT_KEY]) do
-          using_protection(callback) do
-            yield if block_given?
-          end
-        end
+      def execute
+        return unless locksmith.lock(item[LOCK_TIMEOUT_KEY])
+        with_cleanup { yield if block_given? }
       end
 
       private

data/lib/sidekiq_unique_jobs/lock/while_executing_reject.rb

@@ -3,14 +3,10 @@
 module SidekiqUniqueJobs
   class Lock
     class WhileExecutingReject < WhileExecuting
-      def execute(callback)
+      def execute
         return reject unless locksmith.lock(item[LOCK_TIMEOUT_KEY])
 
-        using_protection(callback) do
-          yield if block_given?
-        end
-
-        unlock
+        with_cleanup { yield if block_given? }
       end
 
       # Private below here, keeping public due to testing reasons

data/lib/sidekiq_unique_jobs/lock/while_executing_requeue.rb

@@ -7,12 +7,13 @@ module SidekiqUniqueJobs
         true
       end
 
-      def execute(callback)
+      def execute
         locksmith.lock(item[LOCK_TIMEOUT_KEY], raise: true) do
-          yield
-          callback.call
+          yield if block_given?
         end
 
+        unlock
+
         Sidekiq::Client.push(item) unless locksmith.locked?
       end
     end

data/lib/sidekiq_unique_jobs/options_with_fallback.rb

@@ -34,7 +34,7 @@ module SidekiqUniqueJobs
     end
 
     def lock
-      @lock ||= lock_class.new(item, @redis_pool)
+      @lock ||= lock_class.new(item, after_unlock_hook, @redis_pool)
     end
 
     def lock_class

data/lib/sidekiq_unique_jobs/server/middleware.rb

@@ -11,7 +11,7 @@ module SidekiqUniqueJobs
         @queue        = queue
         return yield if unique_disabled?
 
-        lock.execute(after_unlock_hook) do
+        lock.execute do
           yield
         end
       end
@@ -19,10 +19,6 @@ module SidekiqUniqueJobs
       protected
 
       attr_reader :item
-
-      def after_unlock_hook
-        -> { worker_class.after_unlock if worker_method_defined?(:after_unlock) }
-      end
     end
   end
 end

data/lib/sidekiq_unique_jobs/sidekiq_worker_methods.rb

@@ -19,6 +19,10 @@ module SidekiqUniqueJobs
       @_worker_class ||= worker_class_constantize # rubocop:disable Naming/MemoizedInstanceVariableName
     end
 
+    def after_unlock_hook
+      -> { worker_class.after_unlock if worker_method_defined?(:after_unlock) }
+    end
+
     # Attempt to constantize a string worker_class argument, always
     # failing back to the original argument when the constant can't be found
     #

data/lib/sidekiq_unique_jobs/unique_args.rb

@@ -6,8 +6,6 @@ require 'sidekiq_unique_jobs/normalizer'
 module SidekiqUniqueJobs
   # This class exists to be testable and the entire api should be considered private
   class UniqueArgs
-    CLASS_NAME = 'SidekiqUniqueJobs::UniqueArgs'
-
     include SidekiqUniqueJobs::Logging
     include SidekiqUniqueJobs::SidekiqWorkerMethods
 
@@ -56,7 +54,8 @@ module SidekiqUniqueJobs
     end
 
     def unique_on_all_queues?
-      item[UNIQUE_ON_ALL_QUEUES_KEY] || worker_options[UNIQUE_ON_ALL_QUEUES_KEY]
+      item[UNIQUE_ACROSS_QUEUES_KEY] || worker_options[UNIQUE_ACROSS_QUEUES_KEY] ||
+        item[UNIQUE_ON_ALL_QUEUES_KEY] || worker_options[UNIQUE_ON_ALL_QUEUES_KEY] # TODO: Remove in v 6.1
     end
 
     def unique_across_workers?

data/lib/sidekiq_unique_jobs/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SidekiqUniqueJobs
-  VERSION = '6.0.0.rc3'
+  VERSION = '6.0.0.rc4'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: sidekiq-unique-jobs
 version: !ruby/object:Gem::Version
-  version: 6.0.0.rc3
+  version: 6.0.0.rc4
 platform: ruby
 authors:
 - Mikael Henriksson
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-06-29 00:00:00.000000000 Z
+date: 2018-06-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: concurrent-ruby
@@ -177,6 +177,8 @@ files:
 - ".eslintignore"
 - ".eslintrc"
 - ".fasterer.yml"
+- ".github/ISSUE_TEMPLATE/bug_report.md"
+- ".github/ISSUE_TEMPLATE/feature_request.md"
 - ".gitignore"
 - ".reek"
 - ".rspec"