activejob-uniqueness 0.1.2 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ff1cfb19bc51b7eb8f4040f700650c51a47376e1ec96c84ae72b764227911f0c
-  data.tar.gz: 3813cf68d80e7d56d828e15f179c677e49fe2f171780e25c0a6990b7b93eed76
+  metadata.gz: cd76f286ed7e6482860247e9b98b754dc1b73a381fcd2579d73e469da95d0200
+  data.tar.gz: 866f0949c68745c168217cbc70b10c65799f70aa372902dbd145781726217acd
 SHA512:
-  metadata.gz: b616c4460d8bffe1be31226c3d18d9d3a7197fd714d680ddd636b154934ff648fb5dface28b0099b59ac865a91a5a0cec85a8230ba8d442bda6f831535405b94
-  data.tar.gz: 938eb49268486827b8b946934838af755f7f449fc75e84f2cde8538e49af7e786edc13583a715ad86b2203a58867e324e8e580cc611d522e512ae7ed235ddae1
+  metadata.gz: 5cb9299a82cc40a9fca3df73c771cf553ead9393b11eca505afea5e55fe3023722db67d509528d0886db66e64569ba1caea04622ebde3ca365df1d768f04d2f4
+  data.tar.gz: 205d78f3c41721bce16c16d991204423fe5429b23c1d6fad48eb1a4ac7f775057f7e15192e8d1c32c97fc0c2c4af60a1584d6a666c253df71e1acce9ba02e5fa

data/CHANGELOG.md

@@ -1,14 +1,119 @@
-## Changes Between 0.1.1 and 0.1.2
+# Changelog
+All notable changes to this project will be documented in this file.
 
-Release lock for Sidekiq adapter when all Sidekiq attempts were unsuccessful or job is deleted manually from Sidekiq::Web
-[PR](https://github.com/veeqo/activejob-uniqueness/pull/5) by @vbyno
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
-## Changes Between 0.1.0 and 0.1.1
+## [Unreleased](https://github.com/veeqo/activejob-uniqueness/compare/v0.4.0...HEAD)
 
-Fixed NoMethodError on `Rails.application.eager_load!` in Rails initializer
-```
-NoMethodError: undefined method `unique' for MyJob:Class
-```
-[PR](https://github.com/veeqo/activejob-uniqueness/pull/4)
 
-## Original Release: 0.1.0
+## [0.4.0](https://github.com/veeqo/activejob-uniqueness/compare/v0.3.2...v0.4.0) - 2024-12-07
+
+### Added
+
+- [#86](https://github.com/veeqo/activejob-uniqueness/pull/86) Add Rails 8.0 rc1 support by[@sharshenov](https://github.com/sharshenov)
+- [#78](https://github.com/veeqo/activejob-uniqueness/pull/78) Add on_redis_connection_error config to adjust to new redlock behaviour by[@nduitz](https://github.com/nduitz)
+
+### Changed
+- [#82](https://github.com/veeqo/activejob-uniqueness/pull/82) Optimize bulk unlocking [@sharshenov](https://github.com/sharshenov)
+
+## [0.3.2](https://github.com/veeqo/activejob-uniqueness/compare/v0.3.1...v0.3.2) - 2024-08-16
+
+### Added
+- [#80](https://github.com/veeqo/activejob-uniqueness/pull/80) Add rails 7.2 support by [@viralpraxis](https://github.com/viralpraxis)
+
+### Changed
+- [#74](https://github.com/veeqo/activejob-uniqueness/pull/74) Fix log subscriber by [@shahidkhaliq](https://github.com/shahidkhaliq)
+
+## [0.3.1](https://github.com/veeqo/activejob-uniqueness/compare/v0.3.0...v0.3.1) - 2023-10-30
+
+### Fixed
+
+- [#67](https://github.com/veeqo/activejob-uniqueness/pull/67) Random redis errors on delete_lock by [@laurafeier](https://github.com/laurafeier)
+
+## [0.3.0](https://github.com/veeqo/activejob-uniqueness/compare/v0.2.5...v0.3.0) - 2023-10-20
+
+### Added
+- [#66](https://github.com/veeqo/activejob-uniqueness/pull/66) Activejob 7.1 support by [@laurafeier](https://github.com/laurafeier)
+
+### Changed
+- [#57](https://github.com/veeqo/activejob-uniqueness/pull/57) Upgrade to Redlock 2 & use redis-client by [@bmulholland](https://github.com/bmulholland)
+
+### Removed
+- Support fo Redlock v1 is removed. Switching to `RedisClient` is [a breaking change of Redlock v2](https://github.com/leandromoreira/redlock-rb/blob/main/CHANGELOG.md#200---2023-02-09).
+
+## [0.2.5](https://github.com/veeqo/activejob-uniqueness/compare/v0.2.4...v0.2.5) - 2023-02-01
+
+### Added
+- [#45](https://github.com/veeqo/activejob-uniqueness/pull/45) Add Dependabot for GitHub Actions by [@petergoldstein](https://github.com/petergoldstein)
+- [#51](https://github.com/veeqo/activejob-uniqueness/pull/51) Add support for Sidekiq 7 by [@dwightwatson](https://github.com/dwightwatson)
+- [#52](https://github.com/veeqo/activejob-uniqueness/pull/52) Add Ruby 3.2.0 to the CI matrix by [@petergoldstein](https://github.com/petergoldstein)
+
+### Changed
+- [#46](https://github.com/veeqo/activejob-uniqueness/pull/46) Fix a method name typo in CHANGELOG by [@y-yagi](https://github.com/y-yagi)
+
+## [0.2.4](https://github.com/veeqo/activejob-uniqueness/compare/v0.2.3...v0.2.4) - 2022-06-22
+
+### Added
+- [#43](https://github.com/veeqo/activejob-uniqueness/pull/43) Run rubocop on Github Actions
+- [#44](https://github.com/veeqo/activejob-uniqueness/pull/44) Add ActiveJob::Uniqueness.reset_manager! method to reset lock manager by [@akostadinov](https://github.com/akostadinov)
+
+### Changed
+- [#42](https://github.com/veeqo/activejob-uniqueness/pull/42) Actualize rubies and gems for tests
+
+## [0.2.3](https://github.com/veeqo/activejob-uniqueness/compare/v0.2.2...v0.2.3) - 2022-02-28
+
+### Added
+- [#36](https://github.com/veeqo/activejob-uniqueness/pull/36) Support ActiveJob/Rails 7.0
+- [#37](https://github.com/veeqo/activejob-uniqueness/pull/37) Add Ruby 3.1 to CI by [@petergoldstein](https://github.com/petergoldstein)
+
+## [0.2.2](https://github.com/veeqo/activejob-uniqueness/compare/v0.2.1...v0.2.2) - 2021-10-22
+
+### Added
+- [#32](https://github.com/veeqo/activejob-uniqueness/pull/32) Add ability to set a custom runtime lock key for `:until_and_while_executing` strategy
+
+## [0.2.1](https://github.com/veeqo/activejob-uniqueness/compare/v0.2.0...v0.2.1) - 2021-08-24
+
+### Added
+- [#30](https://github.com/veeqo/activejob-uniqueness/pull/30) Add Sidekiq::JobRecord support (reported by [@dwightwatson](https://github.com/dwightwatson))
+
+## [0.2.0](https://github.com/veeqo/activejob-uniqueness/compare/v0.1.4...v0.2.0) - 2021-05-09
+
+### Added
+- [#22](https://github.com/veeqo/activejob-uniqueness/pull/22) Test with ruby 3.0.1
+
+### Changed
+- [#20](https://github.com/veeqo/activejob-uniqueness/pull/20) **Breaking** Sidekiq patch is not applied automatically anymore
+- [#21](https://github.com/veeqo/activejob-uniqueness/pull/21) Migrate from Travis to Github Actions
+- [#24](https://github.com/veeqo/activejob-uniqueness/pull/24) The default value for `retry_count` of redlock is now 0
+- Require ruby 2.5+
+
+## [0.1.4](https://github.com/veeqo/activejob-uniqueness/compare/v0.1.3...v0.1.4) - 2020-09-22
+
+### Fixed
+- [#11](https://github.com/veeqo/activejob-uniqueness/pull/11) Fix deprecation warnings for ruby 2.7 by [@DanAndreasson](https://github.com/DanAndreasson)
+- [#13](https://github.com/veeqo/activejob-uniqueness/pull/13) Fix deprecation warnings for ruby 2.7
+
+## [0.1.3](https://github.com/veeqo/activejob-uniqueness/compare/v0.1.2...v0.1.3) - 2020-08-17
+
+### Fixed
+- [#7](https://github.com/veeqo/activejob-uniqueness/pull/7) Fix deprecation warnings for ruby 2.7 by [@tonobo](https://github.com/tonobo)
+
+### Changed
+- [#8](https://github.com/veeqo/activejob-uniqueness/pull/8) Use appraisal gem to control gem versions of tests matrix
+- [#9](https://github.com/veeqo/activejob-uniqueness/pull/9) Refactor of Sidekiq API patch. Fixes [#6](https://github.com/veeqo/activejob-uniqueness/issues/6) Rails boot error for version 0.1.2
+- [#10](https://github.com/veeqo/activejob-uniqueness/pull/10) Refactor changelog to comply with Keep a Changelog
+
+## [0.1.2](https://github.com/veeqo/activejob-uniqueness/compare/v0.1.1...v0.1.2) - 2020-07-30
+
+### Added
+-  [#5](https://github.com/veeqo/activejob-uniqueness/pull/5) Release lock for Sidekiq adapter by [@vbyno](https://github.com/vbyno)
+
+## [0.1.1](https://github.com/veeqo/activejob-uniqueness/compare/v0.1.0...v0.1.1) - 2020-07-23
+
+### Fixed
+- [#4](https://github.com/veeqo/activejob-uniqueness/pull/4) Fix `NoMethodError` on `Rails.application.eager_load!` in Rails initializer
+
+## [0.1.0](https://github.com/veeqo/activejob-uniqueness/releases/tag/v0.1.0) - 2020-07-05
+
+### Added
+- Job uniqueness for ActiveJob

data/README.md

@@ -1,5 +1,5 @@
 # Job uniqueness for ActiveJob
-[![Build Status](https://travis-ci.com/veeqo/activejob-uniqueness.svg?branch=master)](https://travis-ci.com/veeqo/activejob-uniqueness) [![Gem Version](https://badge.fury.io/rb/activejob-uniqueness.svg)](https://badge.fury.io/rb/activejob-uniqueness)
+[![Build Status](https://github.com/veeqo/activejob-uniqueness/actions/workflows/main.yml/badge.svg?branch=main)](https://github.com/veeqo/activejob-uniqueness/actions/workflows/main.yml) [![Gem Version](https://badge.fury.io/rb/activejob-uniqueness.svg)](https://badge.fury.io/rb/activejob-uniqueness)
 
 The gem allows to protect job uniqueness with next strategies:
 
@@ -8,10 +8,16 @@ The gem allows to protect job uniqueness with next strategies:
 | `until_executing` | when **pushed** to the queue | when **processing starts** |
 | `until_executed` | when **pushed** to the queue | when the job is **processed successfully** |
 | `until_expired` | when **pushed** to the queue | when the lock is **expired** |
-| `until_and_while_executing` | when **pushed** to the queue | when **processing starts**<br>a runtime lock is acquired to **prevent simultaneous jobs** |
+| `until_and_while_executing` | when **pushed** to the queue | when **processing starts**<br>a runtime lock is acquired to **prevent simultaneous jobs**<br>*has extra options: `runtime_lock_ttl`, `on_runtime_conflict`* |
 | `while_executing` | when **processing starts** | when the job is **processed**<br>with any result including an error |
 
-Inspired by [SidekiqUniqueJobs](https://github.com/mhenrixon/sidekiq-unique-jobs), uses [Redlock](https://github.com/leandromoreira/redlock-rb) under the hood, sponsored by [Veeqo](https://www.veeqo.com/).
+Inspired by [SidekiqUniqueJobs](https://github.com/mhenrixon/sidekiq-unique-jobs), uses [Redlock](https://github.com/leandromoreira/redlock-rb) under the hood.
+
+<p align="center">
+  <a href="https://www.veeqo.com/" title="Sponsored by Veeqo">
+    <img src="https://static.veeqo.com/assets/sponsored_by_veeqo.png" width="360" />
+  </a>
+</p>
 
 ## Installation
 
@@ -21,6 +27,11 @@ Add the `activejob-uniqueness` gem to your Gemfile.
 gem 'activejob-uniqueness'
 ```
 
+If you want jobs unlocking for Sidekiq Web UI, require the patch explicitly. [**Queues cleanup becomes slower!**](#sidekiq-api-support)
+```ruby
+gem 'activejob-uniqueness', require: 'active_job/uniqueness/sidekiq_patch'
+```
+
 And run `bundle install` command.
 
 ## Configuration
@@ -32,29 +43,115 @@ To override the defaults, create an initializer `config/initializers/active_job_
 rails generate active_job:uniqueness:install
 ```
 
+This gem relies on `redlock` for it's Redis connection, that means **it will not inherit global configuration of `Sidekiq`**. To configure the connection, you can use `config.redlock_servers`, for example to disable SSL verification for Redis/Key-Value cloud providers:
+
+```ruby
+ActiveJob::Uniqueness.configure do |config|
+  config.redlock_servers = [
+    RedisClient.new(
+      url: ENV['REDIS_URL'],
+      ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE }
+    )
+  ]
+end
+```
+
 ## Usage
 
-Define uniqueness strategy for your job via `unique` class method:
+
+### Make the job to be unique
 
 ```ruby
 class MyJob < ActiveJob::Base
+  # new jobs with the same args will raise error until existing one is executed
   unique :until_executed
 
-  # Custom expiration:
-  # unique :until_executed, lock_ttl: 3.hours
+  def perform(args)
+    # work
+  end
+end
+```
 
-  # Do not raise error on non unique jobs enqueuing:
-  # unique :until_executed, on_conflict: :log
+### Tune uniqueness settings per job
 
-  # Handle conflict by custom Proc:
-  # unique :until_executed, on_conflict: ->(job) { job.logger.info 'Oops' }
+```ruby
+class MyJob < ActiveJob::Base
+  # new jobs with the same args will be logged within 3 hours or until existing one is being executing
+  unique :until_executing, lock_ttl: 3.hours, on_conflict: :log
 
-  # The :until_and_while_executing strategy supports extra attributes for a runtime lock:
-  # unique :until_and_while_executing runtime_lock_ttl: 10.minutes, on_runtime_conflict: :log
+  def perform(args)
+    # work
+  end
 end
 ```
 
-ActiveJob::Uniqueness allows to manually unlock jobs:
+You can set defaults globally with [the configuration](#configuration)
+
+### Control lock conflicts
+
+```ruby
+class MyJob < ActiveJob::Base
+  # Proc gets the job instance including its arguments
+  unique :until_executing, on_conflict: ->(job) { job.logger.info "Oops: #{job.arguments}" }
+
+  def perform(args)
+    # work
+  end
+end
+```
+
+### Control redis connection errors
+
+```ruby
+class MyJob < ActiveJob::Base
+  # Proc gets the job instance including its arguments, and as keyword arguments the resource(lock key) `resource` and the original error `error`
+  unique :until_executing, on_redis_connection_error: ->(job, resource: _, error: _) { job.logger.info "Oops: #{job.arguments}" }
+
+  def perform(args)
+    # work
+  end
+end
+```
+
+### Control lock key arguments
+
+```ruby
+class MyJob < ActiveJob::Base
+  unique :until_executed
+
+  def perform(foo, bar, baz)
+    # work
+  end
+
+  def lock_key_arguments
+    arguments.first(2) # baz is ignored
+  end
+end
+```
+
+### Control the lock key
+
+```ruby
+class MyJob < ActiveJob::Base
+  unique :until_executed
+
+  def perform(foo, bar, baz)
+    # work
+  end
+
+  def lock_key
+    'qux' # completely custom lock key
+  end
+
+  def runtime_lock_key
+    'quux' # completely custom runtime lock key for :until_and_while_executing
+  end
+end
+```
+
+### Unlock jobs manually
+
+The selected strategy automatically unlocks jobs, but in some cases (e.g. the queue is purged) it is handy to unlock jobs manually.
 
 ```ruby
 # Remove the lock for particular arguments:
@@ -91,7 +188,30 @@ ActiveJob::Uniqueness instruments `ActiveSupport::Notifications` with next event
 
 And then writes to `ActiveJob::Base.logger`.
 
-### ActiveJob prior to version `6.1` will always log `Enqueued MyJob (Job ID) ...` even if the callback chain was halted. [Details](https://github.com/rails/rails/pull/37830)
+**ActiveJob prior to version `6.1` will always log `Enqueued MyJob (Job ID) ...` even if the callback chain is halted. [Details](https://github.com/rails/rails/pull/37830)**
+
+## Testing
+
+Run redis server (in separate console):
+```
+docker run --rm -p 6379:6379 redis
+```
+
+Run tests with:
+
+```sh
+bundle
+rake
+```
+
+## Sidekiq API support
+
+ActiveJob::Uniqueness supports Sidekiq API to unset job locks on queues cleanup (e.g. via Sidekiq Web UI). Starting Sidekiq 5.1 job death also triggers locks cleanup.
+Take into account that **[big queues cleanup becomes much slower](https://github.com/veeqo/activejob-uniqueness/issues/16)** because each job is being unlocked individually. In order to activate Sidekiq API patch require it explicitly in your Gemfile:
+
+```ruby
+gem 'activejob-uniqueness', require: 'active_job/uniqueness/sidekiq_patch'
+```
 
 ## Contributing
 
@@ -100,3 +220,9 @@ Bug reports and pull requests are welcome on GitHub at https://github.com/veeqo/
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+## About [Veeqo](https://www.veeqo.com)
+
+At Veeqo, our team of Engineers is on a mission to create a world-class Inventory and Shipping platform, built to the highest standards in best coding practices. We are a growing team, looking for other passionate developers to [join us](https://veeqo-ltd.breezy.hr/) on our journey. If you're looking for a career working for one of the most exciting tech companies in ecommerce, we want to hear from you.
+
+[Veeqo developers blog](https://devs.veeqo.com)

data/lib/active_job/uniqueness/{patch.rb → active_job_patch.rb}

@@ -16,7 +16,7 @@ module ActiveJob
     #   end
     # end
     #
-    module Patch
+    module ActiveJobPatch
       extend ActiveSupport::Concern
 
       class_methods do
@@ -27,6 +27,7 @@ module ActiveJob
         def unique(strategy, options = {})
           validate_on_conflict_action!(options[:on_conflict])
           validate_on_conflict_action!(options[:on_runtime_conflict])
+          validate_on_redis_connection_error!(options[:on_redis_connection_error])
 
           self.lock_strategy_class = ActiveJob::Uniqueness::Strategies.lookup(strategy)
           self.lock_options = options
@@ -40,7 +41,9 @@ module ActiveJob
 
         private
 
-        delegate :validate_on_conflict_action!, to: :'ActiveJob::Uniqueness.config'
+        delegate :validate_on_conflict_action!,
+                 :validate_on_redis_connection_error!,
+                 to: :'ActiveJob::Uniqueness.config'
       end
 
       included do
@@ -55,7 +58,7 @@ module ActiveJob
       end
 
       def lock_strategy
-        @lock_strategy ||= lock_strategy_class.new(lock_options.merge(lock_key: lock_key, job: self))
+        @lock_strategy ||= lock_strategy_class.new(job: self)
       end
 
       # Override in your job class if you want to customize arguments set for a digest.
@@ -64,21 +67,16 @@ module ActiveJob
       end
 
       # Override lock_key method in your job class if you want to build completely custom lock key.
-      delegate :lock_key, to: :lock_key_generator
+      delegate :lock_key, :runtime_lock_key, to: :lock_key_generator
 
       def lock_key_generator
-        ActiveJob::Uniqueness::LockKey.new job_class_name: self.class.name,
-                                           arguments: lock_key_arguments
+        @lock_key_generator ||= ActiveJob::Uniqueness::LockKey.new job_class_name: self.class.name,
+                                                                   arguments: lock_key_arguments
       end
     end
 
     ActiveSupport.on_load(:active_job) do
-      ActiveJob::Base.include ActiveJob::Uniqueness::Patch
-
-      if ::ActiveJob.const_defined?(:Railtie) &&
-         ActiveJob::Railtie.config.active_job.queue_adapter.to_sym == :sidekiq
-        require_relative 'patch/sidekiq'
-      end
+      ActiveJob::Base.include ActiveJob::Uniqueness::ActiveJobPatch
     end
   end
 end

data/lib/active_job/uniqueness/configuration.rb

@@ -11,14 +11,19 @@ module ActiveJob
     class Configuration
       include ActiveSupport::Configurable
 
-      config_accessor(:lock_ttl) { 1.day }
+      config_accessor(:lock_ttl) { 86_400 } # 1.day
       config_accessor(:lock_prefix) { 'activejob_uniqueness' }
       config_accessor(:on_conflict) { :raise }
-      config_accessor(:digest_method) { OpenSSL::Digest::MD5 }
+      config_accessor(:on_redis_connection_error) { :raise }
       config_accessor(:redlock_servers) { [ENV.fetch('REDIS_URL', 'redis://localhost:6379')] }
-      config_accessor(:redlock_options) { {} }
+      config_accessor(:redlock_options) { { retry_count: 0 } }
       config_accessor(:lock_strategies) { {} }
 
+      config_accessor(:digest_method) do
+        require 'openssl'
+        OpenSSL::Digest::MD5
+      end
+
       def on_conflict=(action)
         validate_on_conflict_action!(action)
 
@@ -30,6 +35,18 @@ module ActiveJob
 
         raise ActiveJob::Uniqueness::InvalidOnConflictAction, "Unexpected '#{action}' action on conflict"
       end
+
+      def on_redis_connection_error=(action)
+        validate_on_redis_connection_error!(action)
+
+        config.on_redis_connection_error = action
+      end
+
+      def validate_on_redis_connection_error!(action)
+        return if action.nil? || action == :raise || action.respond_to?(:call)
+
+        raise ActiveJob::Uniqueness::InvalidOnConflictAction, "Unexpected '#{action}' action on_redis_connection_error"
+      end
     end
   end
 end

data/lib/active_job/uniqueness/lock_key.rb

@@ -28,11 +28,19 @@ module ActiveJob
         ].join(':')
       end
 
+      # used only by :until_and_while_executing strategy
+      def runtime_lock_key
+        [
+          lock_key,
+          'runtime'
+        ].join(':')
+      end
+
       def wildcard_key
         [
           lock_prefix,
           normalized_job_class_name,
-          arguments.any? ? arguments_key_part + '*' : '*'
+          arguments.any? ? "#{arguments_key_part}*" : '*'
         ].compact.join(':')
       end
 

data/lib/active_job/uniqueness/lock_manager.rb

@@ -9,24 +9,40 @@ module ActiveJob
       # Unlocks a resource by resource only.
       def delete_lock(resource)
         @servers.each do |server|
-          server.instance_variable_get(:'@redis').with do |conn|
-            conn.del resource
+          synced_redis_connection(server) do |conn|
+            conn.call('DEL', resource)
           end
         end
 
         true
       end
 
+      DELETE_LOCKS_SCAN_COUNT = 1000
+
       # Unlocks multiple resources by key wildcard.
       def delete_locks(wildcard)
         @servers.each do |server|
-          server.instance_variable_get(:'@redis').with do |conn|
-            conn.scan_each(match: wildcard).each { |key| conn.del key }
+          synced_redis_connection(server) do |conn|
+            cursor = 0
+            while cursor != '0'
+              cursor, keys = conn.call('SCAN', cursor, 'MATCH', wildcard, 'COUNT', DELETE_LOCKS_SCAN_COUNT)
+              conn.call('UNLINK', *keys) unless keys.empty?
+            end
           end
         end
 
         true
       end
+
+      private
+
+      def synced_redis_connection(server, &block)
+        if server.respond_to?(:synchronize)
+          server.synchronize(&block)
+        else
+          server.instance_variable_get(:@redis).with(&block)
+        end
+      end
     end
   end
 end