sidekiq-unique-jobs 7.0.0.beta15 → 7.0.0.beta20

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fd354b14a7f7872a112b42ea75ab268b9781b21f3c116458a1eb38e6d91c4e02
-  data.tar.gz: '02469065fc7048993402dbb9670c0e50e583f5c63d59dd113ea6d3d3bb4f471f'
+  metadata.gz: eb9187151e279ce0c7d654c62b86c57d3ae6ba21639206050943b72032df9f03
+  data.tar.gz: 54aaf4e5179a445ebc551705a68ea177c7cc16c29f5e06d3d8287a683176c58b
 SHA512:
-  metadata.gz: 2c47febf8a69f7738131a3d0bfe1b8560be1b0e4332ce933f421a0314a6a92139c71ef8b18ff249de64c84421ebb1c9cd5ff6b275795545a0abdaacbf4d08320
-  data.tar.gz: fa2378be7428b1973a7c71ffb3abb40a4d76d3ef61ea2c291c4d39790ee623e4d9a5327055b5113149c9db4787106843a6d5c0f111b64eff0677a783e3701bc7
+  metadata.gz: 7c0c34048dd03e862038d92f2b092b03102593899f1c9b30f290dc8ef6a960b67b7583ae82e84d40d69cc001eff1342bb4c7092abfdb6269e8bc8adae2752c79
+  data.tar.gz: 22ea78f67d06a986f3accc92799d83f1c18cba92d4ca2f2a1cebb386d902f6f4105bcd52e4a4235388748c95edca0efa7befe622ed24d25ab509ad4281bc9ced

data/CHANGELOG.md

@@ -1,5 +1,81 @@
 # Changelog
 
+## [Unreleased](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/HEAD)
+
+[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.0.0.beta19...HEAD)
+
+**Fixed bugs:**
+
+- Reaper can't be registered again if sidekiq gets killed by SIGKILL [\#490](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/490)
+
+## [v7.0.0.beta19](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.0.beta19) (2020-05-21)
+
+[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.0.0.beta18...v7.0.0.beta19)
+
+**Fixed bugs:**
+
+- Expire reaper when not checking in [\#508](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/508) ([mhenrixon](https://github.com/mhenrixon))
+
+## [v7.0.0.beta18](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.0.beta18) (2020-05-21)
+
+[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.0.0.beta17...v7.0.0.beta18)
+
+**Fixed bugs:**
+
+- Stringify on\_conflict hash in Job prepare method [\#507](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/507) ([jasonbekolay](https://github.com/jasonbekolay))
+
+## [v7.0.0.beta17](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.0.beta17) (2020-05-20)
+
+[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.0.0.beta16...v7.0.0.beta17)
+
+**Implemented enhancements:**
+
+- Try GitHub actions [\#505](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/505) ([mhenrixon](https://github.com/mhenrixon))
+
+**Fixed bugs:**
+
+- Deep stringify worker options to account for hash in on\_conflict [\#506](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/506) ([jasonbekolay](https://github.com/jasonbekolay))
+
+## [v7.0.0.beta16](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.0.beta16) (2020-05-19)
+
+[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v6.0.22...v7.0.0.beta16)
+
+**Fixed bugs:**
+
+- Deprecate configuration options with `default\_` [\#504](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/504) ([mhenrixon](https://github.com/mhenrixon))
+- Fix access to both server and client conflict [\#503](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/503) ([mhenrixon](https://github.com/mhenrixon))
+
+**Closed issues:**
+
+- V7 Beta 15 `on\_conflict:` with Hash does not work on server [\#499](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/499)
+
+## [v6.0.22](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v6.0.22) (2020-04-13)
+
+[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.0.0.beta15...v6.0.22)
+
+**Fixed bugs:**
+
+- \[LockType\] Allow overwriting default lock type at item level [\#497](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/497) ([EmmN](https://github.com/EmmN))
+
+## [v7.0.0.beta15](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.0.beta15) (2020-04-10)
+
+[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.0.0.beta14...v7.0.0.beta15)
+
+**Implemented enhancements:**
+
+- Duplicated scripts [\#492](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/492)
+- CI: Use jruby-9.2.11.1 [\#485](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/485) ([olleolleolle](https://github.com/olleolleolle))
+
+**Fixed bugs:**
+
+- V7 - `on\_conflict:` no longer accepts a Hash [\#495](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/495)
+- Brpoplpush::RedisScript::LuaError: WRONGTYPE Operation against a key holding the wrong kind of value [\#491](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/491)
+- Lua script bug [\#489](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/489)
+- Reaper will delete locks for running jobs [\#488](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/488)
+- Fix access to hash members [\#496](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/496) ([mhenrixon](https://github.com/mhenrixon))
+- Fix cursor assignment [\#494](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/494) ([mhenrixon](https://github.com/mhenrixon))
+- Prevent reaping of active jobs [\#493](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/493) ([mhenrixon](https://github.com/mhenrixon))
+
 ## [v7.0.0.beta14](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.0.beta14) (2020-03-30)
 
 [Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v6.0.21...v7.0.0.beta14)
@@ -133,6 +209,10 @@
 
 [Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v6.0.18...v7.0.0.beta6)
 
+**Implemented enhancements:**
+
+- Clarify usage with global\_id and sidekiq-status [\#455](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/455) ([mhenrixon](https://github.com/mhenrixon))
+
 **Merged pull requests:**
 
 - Fix that Sidekiq now sends instance of worker [\#459](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/459) ([mhenrixon](https://github.com/mhenrixon))
@@ -175,10 +255,6 @@
 
 [Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v6.0.16...v6.0.17)
 
-**Implemented enhancements:**
-
-- Clarify usage with global\_id and sidekiq-status [\#455](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/455) ([mhenrixon](https://github.com/mhenrixon))
-
 **Fixed bugs:**
 
 - Allow redis namespace to work with deletion [\#451](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/451) ([mhenrixon](https://github.com/mhenrixon))
@@ -473,7 +549,6 @@
 **Fixed bugs:**
 
 - Enable replace strategy [\#315](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/315) ([mhenrixon](https://github.com/mhenrixon))
-- Remove unused method [\#307](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/307) ([mhenrixon](https://github.com/mhenrixon))
 
 **Closed issues:**
 
@@ -499,6 +574,7 @@
 **Fixed bugs:**
 
 - :until\_executed is throwing errors and not requeuing the job. [\#256](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/256)
+- Remove unused method [\#307](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/307) ([mhenrixon](https://github.com/mhenrixon))
 
 **Closed issues:**
 
@@ -840,6 +916,7 @@
 - missed space [\#188](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/188) ([TheBigSadowski](https://github.com/TheBigSadowski))
 - Convert unless if to just 1 if [\#179](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/179) ([otzy007](https://github.com/otzy007))
 - fix for \#168. Handle the NOSCRIPT by sending the script again [\#178](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/178) ([otzy007](https://github.com/otzy007))
+- Fixed gitter badge link [\#176](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/176) ([andrew](https://github.com/andrew))
 
 ## [v4.0.17](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v4.0.17) (2016-03-02)
 
@@ -855,7 +932,6 @@
 
 **Merged pull requests:**
 
-- Fixed gitter badge link [\#176](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/176) ([andrew](https://github.com/andrew))
 - Fix for sidekiq delete failing for version 3.4.x  [\#167](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/167) ([theprogrammerin](https://github.com/theprogrammerin))
 - Run lock timeout configurable [\#164](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/164) ([Slania](https://github.com/Slania))
 

data/lib/sidekiq_unique_jobs/config.rb

@@ -3,8 +3,8 @@
 module SidekiqUniqueJobs
   # ThreadSafe config exists to be able to document the config class without errors
   ThreadSafeConfig = Concurrent::MutableStruct.new("ThreadSafeConfig",
-                                                   :default_lock_timeout,
-                                                   :default_lock_ttl,
+                                                   :lock_timeout,
+                                                   :lock_ttl,
                                                    :enabled,
                                                    :unique_prefix,
                                                    :logger,
@@ -24,6 +24,7 @@ module SidekiqUniqueJobs
   # Shared class for dealing with gem configuration
   #
   # @author Mauro Berlanda <mauro.berlanda@gmail.com>
+  # rubocop:disable Metrics/ClassLength
   class Config < ThreadSafeConfig
     #
     # @return [Hash<Symbol, SidekiqUniqueJobs::Lock::BaseLock] all available queued locks
@@ -183,6 +184,30 @@ module SidekiqUniqueJobs
       )
     end
 
+    def default_lock_ttl=(obj)
+      warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated. Please use `#{class_name}#lock_ttl=` instead."
+      self.lock_ttl = obj
+    end
+
+    def default_lock_timeout=(obj)
+      warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated. Please use `#{class_name}#lock_timeout=` instead."
+      self.lock_timeout = obj
+    end
+
+    def default_lock_ttl
+      warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated. Please use `#{class_name}#lock_ttl` instead."
+      lock_ttl
+    end
+
+    def default_lock_timeout
+      warn "[DEPRECATION] `#{class_name}##{__method__}` is deprecated. Please use `#{class_name}#lock_timeout` instead."
+      lock_timeout
+    end
+
+    def class_name
+      @class_name ||= self.class.name
+    end
+
     #
     # Adds a lock type to the configuration. It will raise if the lock exists already
     #
@@ -234,4 +259,5 @@ module SidekiqUniqueJobs
       current_redis_version
     end
   end
+  # rubocop:enable Metrics/ClassLength
 end

data/lib/sidekiq_unique_jobs/core_ext.rb

@@ -20,6 +20,30 @@ class Hash
     end
   end
 
+  unless {}.respond_to?(:deep_stringify_keys)
+    #
+    # Depp converts all keys to string
+    #
+    #
+    # @return [Hash<String>]
+    #
+    def deep_stringify_keys
+      deep_transform_keys(&:to_s)
+    end
+  end
+
+  unless {}.respond_to?(:deep_transform_keys)
+    #
+    # Deep transfor all keys by yielding to the caller
+    #
+    #
+    # @return [Hash<String>]
+    #
+    def deep_transform_keys(&block)
+      _deep_transform_keys_in_object(self, &block)
+    end
+  end
+
   unless {}.respond_to?(:stringify_keys)
     #
     # Converts all keys to string
@@ -66,6 +90,24 @@ class Hash
       omit
     end
   end
+
+  private
+
+  unless {}.respond_to?(:_deep_transform_keys_in_object)
+    # support methods for deep transforming nested hashes and arrays
+    def _deep_transform_keys_in_object(object, &block)
+      case object
+      when Hash
+        object.each_with_object({}) do |(key, value), result|
+          result[yield(key)] = _deep_transform_keys_in_object(value, &block)
+        end
+      when Array
+        object.map { |element| _deep_transform_keys_in_object(element, &block) }
+      else
+        object
+      end
+    end
+  end
 end
 
 #

data/lib/sidekiq_unique_jobs/job.rb

@@ -10,6 +10,7 @@ module SidekiqUniqueJobs
     # Adds timeout, expiration, lock_args, lock_prefix and lock_digest to the sidekiq job hash
     # @return [Hash] the job hash
     def prepare(item)
+      stringify_on_conflict_hash(item)
       add_lock_timeout(item)
       add_lock_ttl(item)
       add_digest(item)
@@ -27,6 +28,13 @@ module SidekiqUniqueJobs
 
     private
 
+    def stringify_on_conflict_hash(item)
+      on_conflict = item[ON_CONFLICT]
+      return unless on_conflict.is_a?(Hash)
+
+      item[ON_CONFLICT] = on_conflict.deep_stringify_keys
+    end
+
     def add_lock_ttl(item)
       item[LOCK_TTL] = SidekiqUniqueJobs::LockTTL.calculate(item)
     end

data/lib/sidekiq_unique_jobs/lock_args.rb

@@ -63,7 +63,6 @@ module SidekiqUniqueJobs
     # @return [Array] args unfiltered when neither of the above
     def filtered_args
       return args if lock_args_disabled?
-      return args if args.empty?
 
       json_args = Normalizer.jsonify(args)
 

data/lib/sidekiq_unique_jobs/lock_config.rb

@@ -53,7 +53,7 @@ module SidekiqUniqueJobs
     # @return [LockConfig]
     #
     def self.from_worker(options)
-      new(options.stringify_keys)
+      new(options.deep_stringify_keys)
     end
 
     def initialize(job_hash = {})
@@ -107,13 +107,13 @@ module SidekiqUniqueJobs
 
     # the strategy to use as conflict resolution from sidekiq client
     def on_client_conflict
-      @on_client_conflict ||= on_conflict[:client] if on_conflict.is_a?(Hash)
+      @on_client_conflict ||= on_conflict["client"] if on_conflict.is_a?(Hash)
       @on_client_conflict ||= on_conflict
     end
 
     # the strategy to use as conflict resolution from sidekiq server
     def on_server_conflict
-      @on_server_conflict ||= on_conflict[:server] if on_conflict.is_a?(Hash)
+      @on_server_conflict ||= on_conflict["server"] if on_conflict.is_a?(Hash)
       @on_server_conflict ||= on_conflict
     end
   end

data/lib/sidekiq_unique_jobs/lock_timeout.rb

@@ -50,13 +50,13 @@ module SidekiqUniqueJobs
 
     #
     # The configured default_lock_timeout
-    # @see SidekiqUniqueJobs::Config#default_lock_timeout
+    # @see SidekiqUniqueJobs::Config#lock_timeout
     #
     #
     # @return [Integer, nil]
     #
     def default_lock_timeout
-      SidekiqUniqueJobs.config.default_lock_timeout
+      SidekiqUniqueJobs.config.lock_timeout
     end
   end
 end

data/lib/sidekiq_unique_jobs/lock_ttl.rb

@@ -11,7 +11,7 @@ module SidekiqUniqueJobs
 
     #
     # Computes lock ttl from job arguments, sidekiq_options.
-    #   Falls back to {default_lock_ttl}
+    #   Falls back to {SidekiqUniqueJobs::Config#lock_ttl}
     #
     # @note this method takes into consideration the time
     #   until a job is scheduled
@@ -57,7 +57,7 @@ module SidekiqUniqueJobs
 
     #
     # Computes lock ttl from job arguments, sidekiq_options.
-    #   Falls back to {default_lock_ttl}
+    #   Falls back to {SidekiqUniqueJobs::Config#lock_ttl}
     #
     # @note this method takes into consideration the time
     #   until a job is scheduled
@@ -70,19 +70,8 @@ module SidekiqUniqueJobs
       ttl ||= worker_options[LOCK_TTL]
       ttl ||= item[LOCK_EXPIRATION] # TODO: Deprecate at some point
       ttl ||= worker_options[LOCK_EXPIRATION] # TODO: Deprecate at some point
-      ttl ||= default_lock_ttl
+      ttl ||= SidekiqUniqueJobs.config.lock_ttl
       ttl && ttl.to_i + time_until_scheduled
     end
-
-    #
-    # The configured default_lock_ttl
-    # @see SidekiqUniqueJobs::Config#default_lock_ttl
-    #
-    #
-    # @return [Integer, nil]
-    #
-    def default_lock_ttl
-      SidekiqUniqueJobs.config.default_lock_ttl
-    end
   end
 end

data/lib/sidekiq_unique_jobs/orphans/manager.rb

@@ -10,6 +10,8 @@ module SidekiqUniqueJobs
     module Manager
       module_function
 
+      DRIFT_FACTOR = 0.02
+
       include SidekiqUniqueJobs::Connection
       include SidekiqUniqueJobs::Logging
 
@@ -56,6 +58,7 @@ module SidekiqUniqueJobs
         @task ||= Concurrent::TimerTask.new(timer_task_options) do
           with_logging_context do
             redis do |conn|
+              refresh_reaper_mutex
               Orphans::Reaper.call(conn)
             end
           end
@@ -117,7 +120,9 @@ module SidekiqUniqueJobs
       # @return [true, false]
       #
       def registered?
-        redis { |conn| conn.get(UNIQUE_REAPER) }.to_i == 1
+        redis do |conn|
+          conn.get(UNIQUE_REAPER).to_i + drift_reaper_interval > current_timestamp
+        end
       end
 
       def disabled?
@@ -131,7 +136,17 @@ module SidekiqUniqueJobs
       # @return [void]
       #
       def register_reaper_process
-        redis { |conn| conn.set(UNIQUE_REAPER, 1) }
+        redis { |conn| conn.set(UNIQUE_REAPER, current_timestamp, nx: true, ex: drift_reaper_interval) }
+      end
+
+      #
+      # Updates mutex key
+      #
+      #
+      # @return [void]
+      #
+      def refresh_reaper_mutex
+        redis { |conn| conn.set(UNIQUE_REAPER, current_timestamp, ex: drift_reaper_interval) }
       end
 
       #
@@ -143,6 +158,14 @@ module SidekiqUniqueJobs
       def unregister_reaper_process
         redis { |conn| conn.del(UNIQUE_REAPER) }
       end
+
+      def drift_reaper_interval
+        reaper_interval + (reaper_interval * DRIFT_FACTOR).to_i
+      end
+
+      def current_timestamp
+        Time.now.to_i
+      end
     end
   end
 end

data/lib/sidekiq_unique_jobs/sidekiq_unique_jobs.rb

@@ -165,7 +165,8 @@ module SidekiqUniqueJobs
   #
   # This is usually called once at startup of an application
   # @param [Hash] options global gem options
-  # @option options [Integer] :default_lock_timeout (default is 0)
+  # @option options [Integer] :lock_timeout (default is 0)
+  # @option options [Integer] :lock_ttl (default is 0)
   # @option options [true,false] :enabled (default is true)
   # @option options [String] :unique_prefix (default is 'uniquejobs')
   # @option options [Logger] :logger (default is Sidekiq.logger)

data/lib/sidekiq_unique_jobs/sidekiq_worker_methods.rb

@@ -16,7 +16,7 @@ module SidekiqUniqueJobs
     def worker_options
       return {} unless sidekiq_worker_class?
 
-      worker_class.get_sidekiq_options.stringify_keys
+      worker_class.get_sidekiq_options.deep_stringify_keys
     end
 
     # Tests that the

data/lib/sidekiq_unique_jobs/version.rb

@@ -3,5 +3,5 @@
 module SidekiqUniqueJobs
   #
   # @return [String] the current SidekiqUniqueJobs version
-  VERSION = "7.0.0.beta15"
+  VERSION = "7.0.0.beta20"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: sidekiq-unique-jobs
 version: !ruby/object:Gem::Version
-  version: 7.0.0.beta15
+  version: 7.0.0.beta20
 platform: ruby
 authors:
 - Mikael Henriksson
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-04-10 00:00:00.000000000 Z
+date: 2020-06-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: brpoplpush-redis_script
@@ -365,7 +365,18 @@ metadata:
   documentation_uri: https://mhenrixon.github.io/sidekiq-unique-jobs
   source_code_uri: https://github.com/mhenrixon/sidekiq-unique-jobs
   changelog_uri: https://github.com/mhenrixon/sidekiq-unique-jobs/CHANGELOG.md
-post_install_message: 
+post_install_message: |
+  This version deprecated the configuration options:
+    - default_lock_ttl
+    - default_lock_ttl=
+    - default_lock_timeout
+    - default_lock_timeout=
+
+  The new methods to use are:
+    - lock_ttl
+    - lock_ttl=
+    - lock_timeout
+    - lock_timeout=
 rdoc_options: []
 require_paths:
 - lib