sidekiq-unique-jobs 5.0.11 → 6.0.0.rc1

data/README.md

@@ -2,15 +2,28 @@
 
 The missing unique jobs for sidekiq
 
-## Requirements
+# Documentation
+
+This is the documentation for the master branch. You can find the documentation for each release by navigating to its tag: https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v5.0.10. 
 
+Below are links to the latest major versions (4 & 5):
+- [v5.0.10](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v5.0.10)
+- [v4.0.18](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v4.0.18)
 
-See https://github.com/mperham/sidekiq#requirements for what is required. Starting from 5.0.0 only sidekiq >= 4 is supported and support for MRI <=  2.1 is dropped.
+## Requirements
 
-### Version 4 Upgrade instructions
+See https://github.com/mperham/sidekiq#requirements for what is required. Starting from 5.0.0 only sidekiq >= 4 is supported and support for MRI <=  2.1 is dropped. ActiveJob is not supported 
 
 Version 5 requires redis >= 3
 
+### ActiveJob
+
+Due to the simplicity of ActiveJob and the complexity of this game there is no officially supported way of doing ActiveJob. If you want to use uniqueness you should be using sidekiq directly. I know some projects started by using ActiveJob out of ignorance and someone has to do a whole lot of work to migrate the workers to use sidekiq directly... 
+
+If you are in this position and you can't figure it out; I have done such migrations for really big clients before. I am a consultant with a ton of experience on such jobs. My rate is fair and I am easy to get along with.
+
+If that is not an option I apologize. This gem won't support ActiveJob moving forward. It would require monkey patching too much.
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -27,85 +40,107 @@ Or install it yourself as:
 
 ## Locking
 
-Like @mperham mentions on [this wiki page](https://github.com/mperham/sidekiq/wiki/Related-Projects#unique-jobs) it is hard to enforce uniqueness with redis in a distributed redis setting.
+Sidekiq consists of a client and a server. The client is responsible for pushing jobs to the queue and the server is responsible for actually processing the jobs. When the client puts the job to the queue the middleware checks for uniqueness and creates a lock. When the server then processes the job that lock is released.
 
-To make things worse there are many ways of wanting to enforce uniqueness.
+### Options
 
-### While Executing
+#### Lock Expiration
 
-Due to discoverability of the different lock types `unique` sidekiq option it was decided to use the `while_executing` as a default. Most people will note that scheduling any number of jobs with the same arguments is possible.
+This is probably not the configuration option you want... 
 
-```ruby
-sidekiq_options unique: :while_executing
-```
+Since the client and the server are disconnected and not running inside the same process, setting a lock expiration is probably not what you want. Any keys that are used by this gem WILL be removed at the time of the expiration. For jobs that are scheduled in the future the key will expire when that job is scheduled + whatever expiration you have set.
 
-Is to make sure that a job can be scheduled any number of times but only executed a single time per argument provided to the job we call this runtime uniqueness. This is probably most useful for background jobs that are fast to execute. (See mhenrixon/sidekiq-unique-jobs#111 for a great example of when this would be right.) While the job is executing/performing no other jobs can be executed at the same time.
+In previous versions there was a default expiration of 30 minutes which didn't work for a lot of long running jobs. Since version 6 there will be no expiration of any jobs from the default configuration. Please don't use `lock_expiration` unless you really know what you are doing.
 
-The way it currently works is that the jobs can be put on the queue but any succeedent job will wait until the first one finishes. 
+```ruby
+sidekiq_options lock_expiration: nil # default - don't expire keys
+sidekiq_options lock_expiration: 20.days # expire this lock in 20 days
+```
 
-There is an example of this to try it out in the `rails_example` application. Run `foreman start` in the root of the directory and open the url: `localhost:5000/work/duplicate_while_executing`.
+#### Lock Timeout
 
-In the console you should see something like:
+This is the timeout (how long to wait) when creating the lock. By default we don't use a timeout so we won't wait for the lock to be created. If you want it is possible to set this like below.
 
-```
-0:32:24 worker.1 | 2017-04-23T08:32:24.955Z 84404 TID-ougq4thko WhileExecutingWorker JID-400ec51c9523f41cd4a35058 INFO: start
-10:32:24 worker.1 | 2017-04-23T08:32:24.956Z 84404 TID-ougq8csew WhileExecutingWorker JID-8d6d9168368eedaed7f75763 INFO: start
-10:32:24 worker.1 | 2017-04-23T08:32:24.957Z 84404 TID-ougq8crt8 WhileExecutingWorker JID-affcd079094c9b26e8b9ba60 INFO: start
-10:32:24 worker.1 | 2017-04-23T08:32:24.959Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f INFO: start
-10:32:24 worker.1 | 2017-04-23T08:32:24.959Z 84404 TID-ougq4thko WhileExecutingWorker JID-400ec51c9523f41cd4a35058 WhileExecutingWorker INFO: perform(1, 2)
-10:32:34 worker.1 | 2017-04-23T08:32:34.964Z 84404 TID-ougq4thko WhileExecutingWorker JID-400ec51c9523f41cd4a35058 INFO: done: 10.009 sec
-10:32:34 worker.1 | 2017-04-23T08:32:34.965Z 84404 TID-ougq8csew WhileExecutingWorker JID-8d6d9168368eedaed7f75763 WhileExecutingWorker INFO: perform(1, 2)
-10:32:44 worker.1 | 2017-04-23T08:32:44.965Z 84404 TID-ougq8crt8 WhileExecutingWorker JID-affcd079094c9b26e8b9ba60 WhileExecutingWorker INFO: perform(1, 2)
-10:32:44 worker.1 | 2017-04-23T08:32:44.965Z 84404 TID-ougq8csew WhileExecutingWorker JID-8d6d9168368eedaed7f75763 INFO: done: 20.009 sec
-10:32:54 worker.1 | 2017-04-23T08:32:54.970Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f WhileExecutingWorker INFO: perform(1, 2)
-10:32:54 worker.1 | 2017-04-23T08:32:54.969Z 84404 TID-ougq8crt8 WhileExecutingWorker JID-affcd079094c9b26e8b9ba60 INFO: done: 30.012 sec
-10:33:04 worker.1 | 2017-04-23T08:33:04.973Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f INFO: done: 40.014 sec
+```ruby
+sidekiq_options lock_timeout: 0 # default - don't wait at all
+sidekiq_options lock_timeout: 5 # wait 5 seconds
+sidekiq_options lock_timeout: nil # lock indefinitely, this process won't continue until it gets a lock. VERY DANGEROUS!!
 ```
 
+#### 
+
 ### Until Executing
 
+Locks from when the client pushes the job to the queue. Will be unlocked before the server starts processing the job.
+
+**NOTE** this is probably not so good for jobs that shouldn't be running simultaneously (aka slow jobs).
+
 ```ruby
 sidekiq_options unique: :until_executing
 ```
 
-This means that a job can only be scheduled into redis once per whatever the configuration of unique arguments. Any jobs added until the first one of the same arguments has been unlocked will just be dropped. This is what was tripping many people up. They would schedule a job to run in the future and it would be impossible to schedule new jobs with those same arguments even immediately. There was some forth and back between also locking jobs on the scheduled queue and the regular queues but in the end I decided it was best to separate these two features out into different locking mechanisms. I think what most people are after is to be able to lock a job while executing or that seems to be what people are most missing at the moment.
-
 ### Until Executed
 
+Locks from when the client pushes the job to the queue. Will be unlocked when the server has successfully processed the job.
+
+
 ```ruby
 sidekiq_options unique: :until_executed
 ```
 
-This is the combination of the two above. First we lock the job until it executes, then as the job begins executes we keep the lock so that no other jobs with the same arguments can execute at the same time.
-
 ### Until Timeout
 
+Locks from when the client pushes the job to the queue. Will be unlocked when the specified timeout has been reached. 
+
 ```ruby
-sidekiq_options unique: :until_timeout
+sidekiq_options unique: :until_expired
 ```
 
-The job won't be unlocked until the timeout/expiry runs out.
-
 ### Unique Until And While Executing
 
+Locks when the client pushes the job to the queue. The queue will be unlocked when the server starts processing the job. The server then goes on to creating a runtime lock for the job to prevent simultaneous jobs from being executed. As soon as the server starts processing a job, the client can push the same job to the queue.
+
 ```ruby
 sidekiq_options unique: :until_and_while_executing
 ```
 
-This lock is exactly what you would expect. It is considered unique in a way until executing begins and it is locked while executing so what differs from `UntilExecuted`?
+### While Executing
 
-The difference is that this job has two types of uniqueness:
-1. It is unique until execution
-2. It is unique while executing
+With this lock type it is possible to put any number of these jobs on the queue, but as the server pops the job from the queue it will create a lock and then wait until other locks are done processing. It *looks* like multiple jobs are running at the same time but in fact the second job will only be waiting for the first job to finish.
 
-That means it locks for any job with the same arguments to be persisted into redis and just like you would expect it will only ever allow one job of the same unique arguments to run at any given time but as soon as the runtime lock has been acquired the schedule/async lock is released.
+#### NOTE:
 
-### Uniqueness Scope
+Unless this job is configured with a `lock_timeout: nil` or `lock_timeout: > 0` then all jobs that are attempted to be executed will just be dropped without waiting.
+
+```ruby
+sidekiq_options unique: :while_executing, lock_timeout: nil
+```
 
+There is an example of this to try it out in the `rails_example` application. Run `foreman start` in the root of the directory and open the url: `localhost:5000/work/duplicate_while_executing`.
+
+In the console you should see something like:
+
+```
+0:32:24 worker.1 | 2017-04-23T08:32:24.955Z 84404 TID-ougq4thko WhileExecutingWorker JID-400ec51c9523f41cd4a35058 INFO: start
+10:32:24 worker.1 | 2017-04-23T08:32:24.956Z 84404 TID-ougq8csew WhileExecutingWorker JID-8d6d9168368eedaed7f75763 INFO: start
+10:32:24 worker.1 | 2017-04-23T08:32:24.957Z 84404 TID-ougq8crt8 WhileExecutingWorker JID-affcd079094c9b26e8b9ba60 INFO: start
+10:32:24 worker.1 | 2017-04-23T08:32:24.959Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f INFO: start
+10:32:24 worker.1 | 2017-04-23T08:32:24.959Z 84404 TID-ougq4thko WhileExecutingWorker JID-400ec51c9523f41cd4a35058 WhileExecutingWorker INFO: perform(1, 2)
+10:32:34 worker.1 | 2017-04-23T08:32:34.964Z 84404 TID-ougq4thko WhileExecutingWorker JID-400ec51c9523f41cd4a35058 INFO: done: 10.009 sec
+10:32:34 worker.1 | 2017-04-23T08:32:34.965Z 84404 TID-ougq8csew WhileExecutingWorker JID-8d6d9168368eedaed7f75763 WhileExecutingWorker INFO: perform(1, 2)
+10:32:44 worker.1 | 2017-04-23T08:32:44.965Z 84404 TID-ougq8crt8 WhileExecutingWorker JID-affcd079094c9b26e8b9ba60 WhileExecutingWorker INFO: perform(1, 2)
+10:32:44 worker.1 | 2017-04-23T08:32:44.965Z 84404 TID-ougq8csew WhileExecutingWorker JID-8d6d9168368eedaed7f75763 INFO: done: 20.009 sec
+10:32:54 worker.1 | 2017-04-23T08:32:54.970Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f WhileExecutingWorker INFO: perform(1, 2)
+10:32:54 worker.1 | 2017-04-23T08:32:54.969Z 84404 TID-ougq8crt8 WhileExecutingWorker JID-affcd079094c9b26e8b9ba60 INFO: done: 30.012 sec
+10:33:04 worker.1 | 2017-04-23T08:33:04.973Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f INFO: done: 40.014 sec
+```
+
+
+### Uniqueness Scope
 
 - Queue specific locks
-- Across all queues - [spec/jobs/unique_on_all_queues_job.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/jobs/unique_on_all_queues_job.rb)
-- Across all workers - [spec/jobs/unique_across_workers_job.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/spec/jobs/unique_across_workers_job.rb)
+- Across all queues - [examples/unique_on_all_queues_job.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/examples/unique_on_all_queues_job.rb)
+- Across all workers - [examples/unique_across_workers_job.rb](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/examples/unique_across_workers_job.rb)
 - Timed / Scheduled jobs
 
 ## Usage
@@ -121,32 +156,22 @@ should be unique. The job will be unique for the number of seconds configured (d
 or until the job has been completed. Thus, the job will be unique for the shorter of the two.  Note that Sidekiq versions before 3.0 will remove job keys after an hour, which means jobs can remain unique for at most an hour.
 
 If you want the unique job to stick around even after it has been successfully
-processed then just set `unique: :until_timeout`.
+processed then just set `unique: :until_expired`.
 
-You can also control the expiration length of the uniqueness check. If you want to enforce uniqueness over a longer period than the default of 30 minutes then you can pass the number of seconds you want to use to the sidekiq options:
+You can also control the `lock_expiration` of the uniqueness check. If you want to enforce uniqueness over a longer period than the default of 30 minutes then you can pass the number of seconds you want to use to the sidekiq options:
 
 ```ruby
-sidekiq_options unique: :until_timeout, unique_expiration: 120 * 60 # 2 hours
+sidekiq_options unique: :until_expired, lock_expiration: 120 * 60 # 2 hours
 ```
 
 For locking modes (`:while_executing` and `:until_and_while_executing`) you can control the expiration length of the runtime uniqueness. If you want to enforce uniqueness over a longer period than the default of 60 seconds, then you can pass the number of seconds you want to use to the sidekiq options:
 
 ```ruby
-sidekiq_options unique: :while_executing, run_lock_expiration: 2 * 60 # 2 minutes
+sidekiq_options unique: :while_executing, lock_expiration: 2 * 60 # 2 minutes
 ```
 
 Requiring the gem in your gemfile should be sufficient to enable unique jobs.
 
-### Usage with ActiveJob
-
-```ruby
-Sidekiq.default_worker_options = {
-  unique: :until_executing,
-  unique_args: ->(args) { [ args.first.except('job_id') ] }
-}
-```
-
-
 ### Finer Control over Uniqueness
 
 Sometimes it is desired to have a finer control over which arguments are used in determining uniqueness of the job, and others may be _transient_. For this use-case, you need to define either a `unique_args` method, or a ruby proc.
@@ -246,9 +271,6 @@ Start the console with the following command `bundle exec jobs console`.
 #### Remove Unique Keys
 `del '*', 100, false` the dry_run and count parameters are both required. This is to have some type of protection against clearing out all uniqueness.
 
-#### Expire
-`expire` clears the unique hash from expired keys
-
 ### Command Line
 
 `bundle exec jobs` displays help on how to use the unique jobs command line.
@@ -259,23 +281,13 @@ There is a [![Join the chat at https://gitter.im/mhenrixon/sidekiq-unique-jobs](
 
 ## Testing
 
-To enable the testing for `sidekiq-unique-jobs`, add `require 'sidekiq_unique_jobs/testing'` to your testing helper.
+This has been probably the most confusing part of this gem. People get really confused with how unreliable the unique jobs have been. I there for decided to do what Mike is doing for sidekiq enterprise. Read the section about unique jobs.
 
-You can if you want use `gem 'mock_redis'` to prevent sidekiq unique jobs using redis.
-
-See https://github.com/mhenrixon/sidekiq-unique-jobs/tree/master/rails_example/spec/controllers/work_controller_spec.rb for an example of how to configure sidekiq and unique jobs without redis.
-
-If you really don't care about testing uniquness and trust we get that stuff right you can (in newer sidekiq versions) remove the client middleware.
+https://www.dailydrip.com/topics/sidekiq/drips/sidekiq-enterprise-unique-jobs
 
 ```ruby
-describe "Some test" do
-  before(:each) do
-    Sidekiq.configure_client do |config|
-      config.client_middleware do |chain|
-        chain.remove SidekiqUniqueJobs::Client::Middleware
-      end
-    end
-  end
+SidekiqUniqueJobs.configure do |config|
+  config.enabled = !Rails.env.test?
 end
 ```
 

data/examples/another_unique_job.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+# :nocov:
+
+class AnotherUniqueJob
+  include Sidekiq::Worker
+  sidekiq_options queue: :working2, retry: 1, backtrace: 10,
+                  unique: :until_executed
+
+  def perform(args)
+    args
+  end
+end

data/examples/custom_queue_job.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+# :nocov:
+
+class CustomQueueJob
+  include Sidekiq::Worker
+  sidekiq_options queue: :customqueue
+
+  def perform(one, two = 'two')
+    [one, two]
+  end
+end

data/examples/custom_queue_job_with_filter_method.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+# :nocov:
+
+require_relative 'custom_queue_job'
+
+class CustomQueueJobWithFilterMethod < CustomQueueJob
+  sidekiq_options unique: :until_executed, unique_args: :args_filter
+
+  def self.args_filter(args)
+    args.first
+  end
+end

data/examples/custom_queue_job_with_filter_proc.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+# :nocov:
+
+require_relative './custom_queue_job'
+
+class CustomQueueJobWithFilterProc < CustomQueueJob
+  # slightly contrived example of munging args to the
+  # worker and removing a random bit.
+  sidekiq_options unique: :until_expired,
+                  unique_args: (lambda do |args|
+                    options = args.extract_options!
+                    options.delete('random')
+                    args + [options]
+                  end)
+end

data/examples/expiring_job.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+# :nocov:
+
+class ExpiringJob
+  include Sidekiq::Worker
+  sidekiq_options unique: :until_executed, lock_expiration: 10 * 60
+
+  def perform(one, two)
+    [one, two]
+  end
+end

data/examples/inline_worker.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+# :nocov:
+
+class InlineWorker
+  include Sidekiq::Worker
+  sidekiq_options unique: :while_executing, lock_timeout: 5
+
+  def perform(one)
+    TestClass.run(one)
+  end
+end

data/examples/just_a_worker.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+# :nocov:
+
+class JustAWorker
+  include Sidekiq::Worker
+
+  sidekiq_options unique: :until_executed, queue: :testqueue
+
+  def perform(options = {})
+    options
+  end
+end

data/examples/long_running_job.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+# :nocov:
+
+class LongRunningJob
+  include Sidekiq::Worker
+  sidekiq_options queue: :customqueue, retry: true, unique: :until_and_while_executing,
+                  lock_expiration: 7_200, retry_count: 10
+  def perform(one, two)
+    [one, two]
+  end
+end

data/examples/main_job.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+# :nocov:
+
+class MainJob
+  include Sidekiq::Worker
+  sidekiq_options queue: :customqueue, unique: :until_executed,
+                  log_duplicate_payload: true
+
+  def perform(arg)
+    [arg]
+  end
+end

data/examples/my_job.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+# :nocov:
+
+class MyJob
+  include Sidekiq::Worker
+  sidekiq_options queue: :working, retry: 1, backtrace: 10
+
+  def perform(one)
+    [one]
+  end
+end

data/examples/my_unique_job.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+# :nocov:
+
+class MyUniqueJob
+  include Sidekiq::Worker
+  sidekiq_options queue: :customqueue,
+                  retry: true,
+                  retry_count: 10,
+                  unique: :until_executed,
+                  lock_expiration: 7_200
+
+  def perform(one, two)
+    [one, two]
+  end
+end

data/examples/my_unique_job_with_filter_method.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+# :nocov:
+
+class MyUniqueJobWithFilterMethod
+  include Sidekiq::Worker
+  sidekiq_options queue: :customqueue,
+                  retry: true,
+                  backtrace: true,
+                  unique: :until_executed,
+                  unique_args: :filtered_args
+
+  def perform(*)
+    # NO-OP
+  end
+
+  def self.filtered_args(args)
+    options = args.extract_options!
+    [args.first, options['type']]
+  end
+end

data/examples/my_unique_job_with_filter_proc.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+# :nocov:
+
+class MyUniqueJobWithFilterProc
+  include Sidekiq::Worker
+  sidekiq_options queue: :customqueue,
+                  retry: true,
+                  backtrace: true,
+                  unique: :until_executed,
+                  unique_args: (lambda do |args|
+                    options = args.extract_options!
+                    [args.first, options['type']]
+                  end)
+
+  def perform(*)
+    # NO-OP
+  end
+end