resque-unique_by_arity 2.0.2 → 3.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.gitignore +2 -0
- data/Gemfile +2 -2
- data/README.md +56 -9
- data/lib/resque-unique_by_arity.rb +17 -0
- data/lib/resque/plugins/unique_by_arity.rb +35 -4
- data/lib/resque/unique_by_arity.rb +18 -116
- data/lib/resque/unique_by_arity/configuration.rb +44 -26
- data/lib/resque/unique_by_arity/global_configuration.rb +52 -0
- data/lib/resque/unique_by_arity/modulizer.rb +20 -43
- data/lib/resque/unique_by_arity/unique_job.rb +53 -0
- data/lib/resque/unique_by_arity/version.rb +1 -1
- metadata +5 -3
- data/.rspec_status +0 -55
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: bfe4371869efe4deb3aed04f46d2417ab32c82e4fadc7954d6d79a0e9cbbe8d2
|
4
|
+
data.tar.gz: 7b8392ab635f8acc02c6e0d9fa09fe79fd287099ede2f75fd654c43661ab161d
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 8e93042522fa854d08126b163847f5fb670321da013f1e7c20a179311aaf7e0c621855d8998cb465e96df38057a7aaa45357cba36caea1ebc035972a65f17727
|
7
|
+
data.tar.gz: 8a9033f210f024ef2cf99805a97e87137b585dbc6ad82ba58c410b2b6dd7d6021cf2f33af84647a85f4aa9c182ef76830674afa81628c5bbd046625efcb3558e
|
data/.gitignore
CHANGED
data/Gemfile
CHANGED
@@ -2,8 +2,8 @@ source 'https://rubygems.org'
|
|
2
2
|
|
3
3
|
git_source(:github) { |repo_name| "https://github.com/#{repo_name}" }
|
4
4
|
|
5
|
-
gem 'resque-unique_in_queue'
|
6
|
-
gem 'resque-unique_at_runtime'
|
5
|
+
gem 'resque-unique_in_queue', path: '/Users/pboling/Documents/code/intricately/resque-unique_in_queue'
|
6
|
+
gem 'resque-unique_at_runtime', path: '/Users/pboling/Documents/code/intricately/resque-unique_at_runtime'
|
7
7
|
|
8
8
|
group :test do
|
9
9
|
unless ENV['TRAVIS']
|
data/README.md
CHANGED
@@ -1,11 +1,18 @@
|
|
1
1
|
# Resque::UniqueByArity
|
2
2
|
|
3
|
-
Because some jobs have parameters that you do not want to consider for
|
3
|
+
Because some jobs have parameters that you do not want to consider for
|
4
|
+
determination of uniqueness.
|
4
5
|
|
5
6
|
NOTE:
|
6
7
|
|
7
|
-
|
8
|
-
|
8
|
+
I rewrote, and renamed, both `resque_solo` and `resque-lonely_job`, becuase they
|
9
|
+
can't be used together. Why? Their `redis_key` methods directly conflict,
|
10
|
+
among other more subtle issues.
|
11
|
+
|
12
|
+
This gem requires use of my rewritten gems for uniqueness enforcement:
|
13
|
+
|
14
|
+
- [`resque-unique_at_runtime`](https://github.com/pboling/resque-unique_at_runtime)
|
15
|
+
- [`resque-unique_in_queue`](https://github.com/pboling/resque-unique_in_queue)
|
9
16
|
|
10
17
|
| Project | Resque::UniqueByArity |
|
11
18
|
|------------------------ | ----------------------- |
|
@@ -24,11 +31,13 @@ Why? `resque-lonely_job` and `resque_solo` can't be used together, because their
|
|
24
31
|
|
25
32
|
## Important Note
|
26
33
|
|
27
|
-
See `lib/resque/unique_by_arity/configuration.rb` for all config options. Only
|
34
|
+
See `lib/resque/unique_by_arity/configuration.rb` for all config options. Only
|
35
|
+
a smattering of what is available is documented in this README.
|
28
36
|
|
29
37
|
## Most Important Note
|
30
38
|
|
31
|
-
You must configure this gem *after* you define the perform class method in your
|
39
|
+
You must configure this gem *after* you define the perform class method in your
|
40
|
+
job or an error will be raised thanks to `perform` not having been defined yet.
|
32
41
|
|
33
42
|
Example:
|
34
43
|
|
@@ -65,9 +74,44 @@ Or install it yourself as:
|
|
65
74
|
|
66
75
|
## Usage
|
67
76
|
|
77
|
+
This gem will take care to set the class instance variables (similar to the
|
78
|
+
familiar `@queue` class instance variable) that are utilized by
|
79
|
+
`resque-unique_in_queue` and `resque-unique_at_runtime` (default values shown):
|
80
|
+
|
81
|
+
```ruby
|
82
|
+
# For resque-unique_at_runtime
|
83
|
+
@runtime_lock_timeout = 60 * 60 * 24 * 5
|
84
|
+
@runtime_requeue_interval = 1
|
85
|
+
@unique_at_runtime_key_base = 'r-uar'.freeze
|
86
|
+
|
87
|
+
# For resque-unique_in_queue
|
88
|
+
@lock_after_execution_period = 0
|
89
|
+
@ttl = -1
|
90
|
+
@unique_in_queue_key_base = 'r-uiq'.freeze
|
91
|
+
```
|
92
|
+
|
93
|
+
All you need to do is configure this gem accordingly:
|
94
|
+
```ruby
|
95
|
+
include Resque::Plugins::UniqueByArity.new(
|
96
|
+
arity_for_uniqueness: 3,
|
97
|
+
unique_at_runtime: true,
|
98
|
+
unique_in_queue: true,
|
99
|
+
# No need to do the following if keeping default values
|
100
|
+
runtime_lock_timeout: 60 * 60 * 24 * 5,
|
101
|
+
runtime_requeue_interval: 1,
|
102
|
+
unique_at_runtime_key_base: 'r-uar'.freeze,
|
103
|
+
lock_after_execution_period: 0,
|
104
|
+
ttl: 0,
|
105
|
+
unique_in_queue_key_base: 'r-uiq'.freeze
|
106
|
+
)
|
107
|
+
```
|
108
|
+
|
68
109
|
### Arity For Uniqueness
|
69
110
|
|
70
|
-
Some jobs have parameters that you do not want to consider for determination of
|
111
|
+
Some jobs have parameters that you do not want to consider for determination of
|
112
|
+
uniqueness. Resque jobs should use simple parameters, **not named parameters**,
|
113
|
+
so you can just specify the number of parameters, counting from the left, you
|
114
|
+
want to be considered for uniqueness.
|
71
115
|
|
72
116
|
```ruby
|
73
117
|
class MyJob
|
@@ -105,7 +149,8 @@ end
|
|
105
149
|
|
106
150
|
### Lock After Execution
|
107
151
|
|
108
|
-
Give the job a break after it finishes running, and don't allow another of the
|
152
|
+
Give the job a break after it finishes running, and don't allow another of the
|
153
|
+
same, with matching args @ configured arity, to start within X seconds.
|
109
154
|
|
110
155
|
```ruby
|
111
156
|
class MyJob
|
@@ -122,7 +167,8 @@ end
|
|
122
167
|
|
123
168
|
### Runtime Lock Timeout
|
124
169
|
|
125
|
-
If runtime lock keys get stale, they will expire on their own after some period.
|
170
|
+
If runtime lock keys get stale, they will expire on their own after some period.
|
171
|
+
You can set the expiration period on a per class basis.
|
126
172
|
|
127
173
|
```ruby
|
128
174
|
class MyJob
|
@@ -263,7 +309,8 @@ end
|
|
263
309
|
|
264
310
|
### Debugging
|
265
311
|
|
266
|
-
Run your worker with `RESQUE_DEBUG=true` to see payloads printed before they are
|
312
|
+
Run your worker with `RESQUE_DEBUG=true` to see payloads printed before they are
|
313
|
+
used to determine uniqueness, as well as a lot of other debugging output.
|
267
314
|
|
268
315
|
### Customize Unique Keys Per Job
|
269
316
|
|
@@ -0,0 +1,17 @@
|
|
1
|
+
require 'resque/unique_by_arity/version'
|
2
|
+
|
3
|
+
# External Gems
|
4
|
+
require 'colorized_string'
|
5
|
+
require 'resque'
|
6
|
+
|
7
|
+
# External Resque Plugins
|
8
|
+
require 'resque-unique_in_queue'
|
9
|
+
require 'resque-unique_at_runtime'
|
10
|
+
|
11
|
+
require 'resque/unique_by_arity/configuration'
|
12
|
+
require 'resque/unique_by_arity/global_configuration'
|
13
|
+
require 'resque/unique_by_arity'
|
14
|
+
require 'resque/unique_by_arity/modulizer'
|
15
|
+
require 'resque/unique_by_arity/unique_job'
|
16
|
+
require 'resque/unique_by_arity/validation'
|
17
|
+
require 'resque/plugins/unique_by_arity'
|
@@ -35,17 +35,48 @@ module Resque
|
|
35
35
|
# As a result we can configure per class.
|
36
36
|
@configuration.base_klass_name = base.to_s
|
37
37
|
@configuration.validate
|
38
|
-
base.send(:extend, Resque::UniqueByArity)
|
38
|
+
base.send(:extend, Resque::UniqueByArity::UniqueJob)
|
39
39
|
base.uniqueness_config_reset(@configuration.dup)
|
40
40
|
|
41
|
-
# gem is unique_in_queue, which is a rewrite of resque-loner
|
42
|
-
# see: https://github.com/
|
41
|
+
# gem is resque-unique_in_queue, which is a rewrite of resque-solo / resque-loner
|
42
|
+
# see: https://github.com/pboling/resque-unique_in_queue
|
43
43
|
# defines a redis_key method, which we have to override.
|
44
44
|
base.send(:include, Resque::Plugins::UniqueInQueue) if @configuration.unique_in_queue || @configuration.unique_across_queues
|
45
45
|
|
46
46
|
# gem is resque-unique_at_runtime, which is a rewrite of resque-lonely_job
|
47
47
|
# see: https://github.com/pboling/resque-unique_at_runtime
|
48
|
-
base.send(:
|
48
|
+
base.send(:include, Resque::Plugins::UniqueAtRuntime) if @configuration.unique_at_runtime
|
49
|
+
|
50
|
+
# For resque-unique_at_runtime
|
51
|
+
#
|
52
|
+
if @configuration.runtime_lock_timeout
|
53
|
+
base.instance_variable_set(:@runtime_lock_timeout, @configuration.runtime_lock_timeout)
|
54
|
+
end
|
55
|
+
|
56
|
+
if @configuration.runtime_requeue_interval
|
57
|
+
base.instance_variable_set(:@runtime_requeue_interval, @configuration.runtime_requeue_interval)
|
58
|
+
end
|
59
|
+
|
60
|
+
if @configuration.unique_at_runtime_key_base
|
61
|
+
base.instance_variable_set(:@unique_at_runtime_key_base, @configuration.unique_at_runtime_key_base)
|
62
|
+
end
|
63
|
+
|
64
|
+
# For resque-unique_in_queue
|
65
|
+
#
|
66
|
+
if @configuration.lock_after_execution_period
|
67
|
+
base.instance_variable_set(:@lock_after_execution_period, @configuration.lock_after_execution_period)
|
68
|
+
end
|
69
|
+
|
70
|
+
if @configuration.ttl
|
71
|
+
base.instance_variable_set(:@ttl, @configuration.ttl)
|
72
|
+
end
|
73
|
+
|
74
|
+
# Normally doesn't make sense to override per each class because
|
75
|
+
# it wouldn't be able to determine or enforce uniqueness across queues,
|
76
|
+
# and general cleanup of stray keys would be nearly impossible.
|
77
|
+
if @configuration.unique_in_queue_key_base
|
78
|
+
base.instance_variable_set(:@unique_in_queue_key_base, @configuration.unique_in_queue_key_base)
|
79
|
+
end
|
49
80
|
|
50
81
|
uniqueness_cop_module = Resque::UniqueByArity::Modulizer.to_mod(@configuration)
|
51
82
|
# This will override methods from both plugins above, if configured for both
|
@@ -1,132 +1,34 @@
|
|
1
|
-
require 'resque/unique_by_arity
|
1
|
+
# Just in case some code still does require 'resque/unique_by_arity'
|
2
|
+
require 'resque-unique_by_arity'
|
2
3
|
|
3
|
-
# External Gems
|
4
|
-
require 'colorized_string'
|
5
|
-
require 'resque'
|
6
|
-
|
7
|
-
# External Resque Plugins
|
8
|
-
require 'resque-unique_in_queue'
|
9
|
-
require 'resque-unique_at_runtime'
|
10
|
-
|
11
|
-
require 'resque/plugins/unique_by_arity'
|
12
|
-
require 'resque/unique_by_arity/configuration'
|
13
|
-
require 'resque/unique_by_arity/modulizer'
|
14
|
-
require 'resque/unique_by_arity/validation'
|
15
|
-
|
16
|
-
# Usage:
|
17
|
-
#
|
18
|
-
# class MyJob
|
19
|
-
# def self.perform(arg1, arg2)
|
20
|
-
# end
|
21
|
-
# include Resque::Plugins::UniqueByArity.new(
|
22
|
-
# arity_for_uniqueness: 1,
|
23
|
-
# arity_validation: :warning, # or nil, false, or :error
|
24
|
-
# unique_at_runtime: true,
|
25
|
-
# unique_in_queue: true
|
26
|
-
# )
|
27
|
-
# end
|
28
|
-
#
|
29
|
-
# NOTE: DO NOT include this module directly.
|
30
|
-
# Use the Resque::Plugins::UniqueByArity approach as above.
|
31
|
-
# This module is ultimately extended into the job class.
|
32
4
|
module Resque
|
33
5
|
module UniqueByArity
|
34
6
|
PLUGIN_TAG = (ColorizedString['[R-UBA] '].green).freeze
|
35
7
|
|
36
|
-
def
|
37
|
-
config_proxy ||=
|
38
|
-
config_proxy.
|
39
|
-
end
|
40
|
-
|
41
|
-
def unique_debug(message, config_proxy = nil)
|
42
|
-
config_proxy ||= uniqueness_configuration
|
43
|
-
config_proxy.unique_logger&.debug("#{Resque::UniqueByArity::PLUGIN_TAG}#{message}") if config_proxy.debug_mode
|
8
|
+
def log(message, config_proxy = nil)
|
9
|
+
config_proxy ||= configuration
|
10
|
+
config_proxy.logger&.send(config_proxy.log_level, message) if config_proxy.logger
|
44
11
|
end
|
45
|
-
module_function(:
|
12
|
+
module_function(:log)
|
46
13
|
|
47
|
-
|
48
|
-
|
49
|
-
|
50
|
-
@uniqueness_configuration
|
14
|
+
def debug(message, config_proxy = nil)
|
15
|
+
config_proxy ||= configuration
|
16
|
+
config_proxy.logger&.debug("#{Resque::UniqueByArity::PLUGIN_TAG}#{message}") if config_proxy.debug_mode
|
51
17
|
end
|
18
|
+
module_function(:debug)
|
52
19
|
|
53
|
-
# For
|
54
|
-
def
|
55
|
-
@
|
56
|
-
yield(@uniqueness_configuration)
|
20
|
+
# For config with a block
|
21
|
+
def configure
|
22
|
+
yield(@configuration)
|
57
23
|
end
|
58
24
|
|
59
25
|
#### CONFIG ####
|
26
|
+
# Access globally configured settings:
|
27
|
+
# >> Resque::UniqueByArity.configuration.logger
|
28
|
+
# => the Logger instance
|
60
29
|
class << self
|
61
|
-
attr_accessor :
|
62
|
-
end
|
63
|
-
def uniqueness_config_reset(config = Configuration.new)
|
64
|
-
@uniqueness_configuration = config
|
65
|
-
end
|
66
|
-
|
67
|
-
def uniqueness_log_level
|
68
|
-
@uniqueness_configuration.log_level
|
69
|
-
end
|
70
|
-
|
71
|
-
def uniqueness_log_level=(log_level)
|
72
|
-
@uniqueness_configuration.log_level = log_level
|
73
|
-
end
|
74
|
-
|
75
|
-
def uniqueness_arity_for_uniqueness
|
76
|
-
@uniqueness_configuration.arity_for_uniqueness
|
77
|
-
end
|
78
|
-
|
79
|
-
def uniqueness_arity_for_uniqueness=(arity_for_uniqueness)
|
80
|
-
@uniqueness_configuration.arity_for_uniqueness = arity_for_uniqueness
|
81
|
-
end
|
82
|
-
|
83
|
-
def uniqueness_arity_validation
|
84
|
-
@uniqueness_configuration.arity_validation
|
85
|
-
end
|
86
|
-
|
87
|
-
def uniqueness_arity_validation=(arity_validation)
|
88
|
-
@uniqueness_configuration.arity_validation = arity_validation
|
89
|
-
end
|
90
|
-
|
91
|
-
def uniqueness_lock_after_execution_period
|
92
|
-
@uniqueness_configuration.lock_after_execution_period
|
93
|
-
end
|
94
|
-
|
95
|
-
def uniqueness_lock_after_execution_period=(lock_after_execution_period)
|
96
|
-
@uniqueness_configuration.lock_after_execution_period = lock_after_execution_period
|
97
|
-
end
|
98
|
-
|
99
|
-
def uniqueness_runtime_lock_timeout
|
100
|
-
@uniqueness_configuration.runtime_lock_timeout
|
101
|
-
end
|
102
|
-
|
103
|
-
def uniqueness_runtime_lock_timeout=(runtime_lock_timeout)
|
104
|
-
@uniqueness_configuration.runtime_lock_timeout = runtime_lock_timeout
|
105
|
-
end
|
106
|
-
|
107
|
-
def uniqueness_unique_at_runtime
|
108
|
-
@uniqueness_configuration.unique_at_runtime
|
109
|
-
end
|
110
|
-
|
111
|
-
def uniqueness_unique_at_runtime=(unique_at_runtime)
|
112
|
-
@uniqueness_configuration.unique_at_runtime = unique_at_runtime
|
113
|
-
end
|
114
|
-
|
115
|
-
def uniqueness_unique_in_queue
|
116
|
-
@uniqueness_configuration.unique_in_queue
|
117
|
-
end
|
118
|
-
|
119
|
-
def uniqueness_unique_in_queue=(unique_in_queue)
|
120
|
-
@uniqueness_configuration.unique_in_queue = unique_in_queue
|
121
|
-
end
|
122
|
-
|
123
|
-
def uniqueness_unique_across_queues
|
124
|
-
@uniqueness_configuration.unique_across_queues
|
125
|
-
end
|
126
|
-
|
127
|
-
def uniqueness_unique_across_queues=(unique_across_queues)
|
128
|
-
@uniqueness_configuration.unique_across_queues = unique_across_queues
|
30
|
+
attr_accessor :configuration
|
129
31
|
end
|
130
|
-
self.
|
32
|
+
self.configuration = GlobalConfiguration.instance # setup defaults
|
131
33
|
end
|
132
34
|
end
|
@@ -1,11 +1,13 @@
|
|
1
1
|
require 'logger'
|
2
2
|
module Resque
|
3
3
|
module UniqueByArity
|
4
|
+
# This class is for configurations that are per job class, *not* app-wide.
|
5
|
+
# Each setting will default to the global config values.
|
4
6
|
class Configuration
|
5
7
|
VALID_ARITY_VALIDATION_LEVELS = [:warning, :error, :skip, nil, false].freeze
|
6
8
|
SKIPPED_ARITY_VALIDATION_LEVELS = [:skip, nil, false].freeze
|
7
|
-
|
8
|
-
|
9
|
+
DEFAULT_LOG_LEVEL = :debug
|
10
|
+
|
9
11
|
attr_accessor :logger
|
10
12
|
attr_accessor :log_level
|
11
13
|
attr_accessor :arity_for_uniqueness
|
@@ -18,27 +20,35 @@ module Resque
|
|
18
20
|
attr_accessor :unique_in_queue
|
19
21
|
attr_accessor :unique_in_queue_key_base
|
20
22
|
attr_accessor :unique_across_queues
|
23
|
+
attr_accessor :ttl
|
21
24
|
attr_accessor :base_klass_name
|
22
25
|
attr_accessor :debug_mode
|
26
|
+
|
23
27
|
def initialize(**options)
|
24
|
-
@logger = options.key?(:logger) ? options[:logger] : Logger.new(STDOUT)
|
25
|
-
@log_level = options.key?(:log_level) ? options[:log_level] : :debug
|
26
|
-
@arity_for_uniqueness = options.key?(:arity_for_uniqueness) ? options[:arity_for_uniqueness] : 1
|
27
|
-
@arity_validation = options.key?(:arity_validation) ? options[:arity_validation] : :warning
|
28
|
+
@logger = options.key?(:logger) ? options[:logger] : defcon(:logger) || Logger.new(STDOUT)
|
29
|
+
@log_level = options.key?(:log_level) ? options[:log_level] : defcon(:log_level) || :debug
|
30
|
+
@arity_for_uniqueness = options.key?(:arity_for_uniqueness) ? options[:arity_for_uniqueness] : defcon(:arity_for_uniqueness) || 1
|
31
|
+
@arity_validation = options.key?(:arity_validation) ? options[:arity_validation] : defcon(:arity_validation) || :warning
|
28
32
|
raise ArgumentError, "Resque::Plugins::UniqueByArity.new requires arity_validation values of #{arity_validation.inspect}, or a class inheriting from Exception, but the value is #{@arity_validation} (#{@arity_validation.class})" unless VALID_ARITY_VALIDATION_LEVELS.include?(@arity_validation) || !@arity_validation.respond_to?(:ancestors) || @arity_validation.ancestors.include?(Exception)
|
29
33
|
|
30
|
-
@
|
31
|
-
@
|
32
|
-
@
|
33
|
-
@
|
34
|
-
@
|
35
|
-
@
|
36
|
-
@
|
37
|
-
@
|
38
|
-
|
34
|
+
@ttl = options.key?(:ttl) ? options[:ttl] : defcon(:ttl) || nil
|
35
|
+
@lock_after_execution_period = options.key?(:lock_after_execution_period) ? options[:lock_after_execution_period] : defcon(:lock_after_execution_period) || nil
|
36
|
+
@runtime_lock_timeout = options.key?(:runtime_lock_timeout) ? options[:runtime_lock_timeout] : defcon(:runtime_lock_timeout) || nil
|
37
|
+
@runtime_requeue_interval = options.key?(:runtime_requeue_interval) ? options[:runtime_requeue_interval] : defcon(:runtime_requeue_interval) || nil
|
38
|
+
@unique_at_runtime = options.key?(:unique_at_runtime) ? options[:unique_at_runtime] : defcon(:unique_at_runtime) || false
|
39
|
+
@unique_at_runtime_key_base = options.key?(:unique_at_runtime_key_base) ? options[:unique_at_runtime_key_base] : defcon(:unique_at_runtime_key_base) || nil
|
40
|
+
@unique_in_queue_key_base = options.key?(:unique_in_queue_key_base) ? options[:unique_in_queue_key_base] : defcon(:unique_in_queue_key_base) || nil
|
41
|
+
@unique_in_queue = options.key?(:unique_in_queue) ? options[:unique_in_queue] : defcon(:unique_in_queue) || false
|
42
|
+
@unique_across_queues = options.key?(:unique_across_queues) ? options[:unique_across_queues] : defcon(:unique_across_queues) || false
|
43
|
+
# Can't be both unique in queue and unique across queues, since they
|
44
|
+
# must necessarily use different locking key structures, and therefore
|
45
|
+
# can't both be active.
|
39
46
|
raise ArgumentError, "Resque::Plugins::UniqueByArity.new requires either one or none of @unique_across_queues and @unique_in_queue to be true. Having both set to true is non-sensical." if @unique_in_queue && @unique_across_queues
|
40
|
-
|
41
|
-
@debug_mode
|
47
|
+
@debug_mode = !!(options.key?(:debug_mode) ? options[:debug_mode] : defcon(:debug_mode))
|
48
|
+
if @debug_mode
|
49
|
+
# Make sure there is a logger when in debug_mode
|
50
|
+
@logger ||= Logger.new(STDOUT)
|
51
|
+
end
|
42
52
|
end
|
43
53
|
|
44
54
|
def validate
|
@@ -52,16 +62,8 @@ module Resque
|
|
52
62
|
end
|
53
63
|
end
|
54
64
|
|
55
|
-
def unique_logger
|
56
|
-
logger
|
57
|
-
end
|
58
|
-
|
59
|
-
def unique_log_level
|
60
|
-
log_level
|
61
|
-
end
|
62
|
-
|
63
65
|
def log(msg)
|
64
|
-
Resque::UniqueByArity.
|
66
|
+
Resque::UniqueByArity.log(msg, self)
|
65
67
|
end
|
66
68
|
|
67
69
|
def to_hash
|
@@ -74,6 +76,7 @@ module Resque
|
|
74
76
|
debug_mode: debug_mode,
|
75
77
|
lock_after_execution_period: lock_after_execution_period,
|
76
78
|
runtime_lock_timeout: runtime_lock_timeout,
|
79
|
+
ttl: ttl,
|
77
80
|
unique_at_runtime: unique_at_runtime,
|
78
81
|
unique_in_queue: unique_in_queue,
|
79
82
|
unique_across_queues: unique_across_queues
|
@@ -121,6 +124,21 @@ module Resque
|
|
121
124
|
end
|
122
125
|
end
|
123
126
|
end
|
127
|
+
|
128
|
+
def defcon(sym)
|
129
|
+
Resque::UniqueByArity.configuration.send(sym)
|
130
|
+
end
|
131
|
+
|
132
|
+
def debug_mode=(val)
|
133
|
+
@debug_mode = !!val
|
134
|
+
end
|
135
|
+
|
136
|
+
private
|
137
|
+
|
138
|
+
def debug_mode_from_env
|
139
|
+
env_debug = ENV['RESQUE_DEBUG']
|
140
|
+
@debug_mode = !!(env_debug == 'true' || (env_debug.is_a?(String) && env_debug.match?(/queue/)))
|
141
|
+
end
|
124
142
|
end
|
125
143
|
end
|
126
144
|
end
|
@@ -0,0 +1,52 @@
|
|
1
|
+
require 'logger'
|
2
|
+
module Resque
|
3
|
+
module UniqueByArity
|
4
|
+
# This class is for configurations that are app-wide, *not per job class.
|
5
|
+
# For this reason it is a Singleton.
|
6
|
+
# Will be used as the default settings for the per-job configs.
|
7
|
+
class GlobalConfiguration < Configuration
|
8
|
+
DEFAULT_LOG_LEVEL = :debug
|
9
|
+
DEFAULT_AT_RUNTIME_KEY_BASE = 'r-uar'.freeze
|
10
|
+
DEFAULT_IN_QUEUE_KEY_BASE = 'r-uiq'.freeze
|
11
|
+
|
12
|
+
# For resque-unique_at_runtime
|
13
|
+
DEFAULT_LOCK_TIMEOUT = 60 * 60 * 24 * 5
|
14
|
+
DEFAULT_REQUEUE_INTERVAL = 1
|
15
|
+
DEFAULT_UNIQUE_AT_RUNTIME_KEY_BASE = 'r-uar'.freeze
|
16
|
+
|
17
|
+
# For resque-unique_in_queue
|
18
|
+
DEFAULT_LOCK_AFTER_EXECUTION_PERIOD = 0
|
19
|
+
DEFAULT_TTL = -1
|
20
|
+
DEFAULT_UNIQUE_IN_QUEUE_KEY_BASE = 'r-uiq'.freeze
|
21
|
+
|
22
|
+
include Singleton
|
23
|
+
|
24
|
+
def initialize
|
25
|
+
debug_mode_from_env
|
26
|
+
@logger = nil
|
27
|
+
@log_level = DEFAULT_LOG_LEVEL
|
28
|
+
@arity_for_uniqueness = nil
|
29
|
+
@arity_validation = nil
|
30
|
+
@lock_after_execution_period = DEFAULT_LOCK_AFTER_EXECUTION_PERIOD
|
31
|
+
@runtime_lock_timeout = DEFAULT_LOCK_TIMEOUT
|
32
|
+
@runtime_requeue_interval = DEFAULT_REQUEUE_INTERVAL
|
33
|
+
@unique_at_runtime_key_base = DEFAULT_AT_RUNTIME_KEY_BASE
|
34
|
+
@unique_in_queue_key_base = DEFAULT_IN_QUEUE_KEY_BASE
|
35
|
+
@unique_at_runtime = false
|
36
|
+
@unique_at_runtime_key_base = DEFAULT_UNIQUE_AT_RUNTIME_KEY_BASE
|
37
|
+
@unique_in_queue_key_base = DEFAULT_UNIQUE_IN_QUEUE_KEY_BASE
|
38
|
+
@unique_in_queue = false
|
39
|
+
@unique_across_queues = false
|
40
|
+
@ttl = DEFAULT_TTL
|
41
|
+
if @debug_mode
|
42
|
+
# Make sure there is a logger when in debug_mode
|
43
|
+
@logger ||= Logger.new(STDOUT)
|
44
|
+
end
|
45
|
+
end
|
46
|
+
|
47
|
+
def defcon(sym)
|
48
|
+
self.send(sym)
|
49
|
+
end
|
50
|
+
end
|
51
|
+
end
|
52
|
+
end
|
@@ -7,7 +7,7 @@ module Resque
|
|
7
7
|
# @return [Array<String, arguments>] the key base hash used to enforce uniqueness, and the arguments from the payload used to calculate it
|
8
8
|
define_method(:redis_unique_hash) do |payload|
|
9
9
|
payload = Resque.decode(Resque.encode(payload))
|
10
|
-
Resque::UniqueByArity.
|
10
|
+
Resque::UniqueByArity.debug("payload is #{payload.inspect}")
|
11
11
|
job = payload['class']
|
12
12
|
# It seems possible that some jobs may not have an "args" key in the payload.
|
13
13
|
args = payload['args'] || []
|
@@ -28,36 +28,13 @@ module Resque
|
|
28
28
|
end
|
29
29
|
end
|
30
30
|
|
31
|
-
if configuration.lock_after_execution_period
|
32
|
-
instance_variable_set(:@lock_after_execution_period, configuration.lock_after_execution_period)
|
33
|
-
end
|
34
|
-
|
35
|
-
if configuration.runtime_lock_timeout
|
36
|
-
instance_variable_set(:@runtime_lock_timeout, configuration.runtime_lock_timeout)
|
37
|
-
end
|
38
|
-
|
39
|
-
if configuration.runtime_requeue_interval
|
40
|
-
instance_variable_set(:@runtime_requeue_interval, configuration.runtime_requeue_interval)
|
41
|
-
end
|
42
|
-
|
43
|
-
if configuration.unique_at_runtime_key_base
|
44
|
-
instance_variable_set(:@unique_at_runtime_key_base, configuration.unique_at_runtime_key_base)
|
45
|
-
end
|
46
|
-
|
47
|
-
if configuration.unique_in_queue_key_base
|
48
|
-
# Can't be overridden per each class because it wouldn't make sense.
|
49
|
-
# It wouldn't be able to determine or enforce uniqueness across queues,
|
50
|
-
# and general cleanup of stray keys would be nearly impossible.
|
51
|
-
Resque::UniqueInQueue.uniq_config&.unique_in_queue_key_base = configuration.unique_in_queue_key_base
|
52
|
-
end
|
53
|
-
|
54
31
|
if configuration.unique_in_queue || configuration.unique_across_queues
|
55
|
-
### Gem: unique_in_queue
|
32
|
+
### Gem: resque-unique_in_queue
|
56
33
|
### Plugin Name: Resque::Plugins::UniqueJob
|
57
34
|
### Provides: Queue-time uniqueness for a single queue, or across queues
|
58
35
|
#
|
59
36
|
# Returns a string, used by Resque::Plugins::UniqueJob, that will be used as the prefix to the redis key
|
60
|
-
define_method(:
|
37
|
+
define_method(:unique_in_queue_redis_key_prefix) do
|
61
38
|
"unique_job:#{self}"
|
62
39
|
end
|
63
40
|
#
|
@@ -69,28 +46,29 @@ module Resque
|
|
69
46
|
# @return [String] the key used to enforce uniqueness (at queue-time)
|
70
47
|
define_method(:unique_in_queue_redis_key) do |queue, payload|
|
71
48
|
unique_hash, args_for_uniqueness = redis_unique_hash(payload)
|
72
|
-
key = "#{
|
73
|
-
Resque::UniqueByArity.
|
49
|
+
key = "#{unique_in_queue_key_namespace(queue)}:#{unique_in_queue_redis_key_prefix}:#{unique_hash}"
|
50
|
+
Resque::UniqueByArity.debug("#{self}.unique_in_queue_redis_key for #{args_for_uniqueness} is: #{ColorizedString[key].green}")
|
74
51
|
key
|
75
52
|
end
|
76
53
|
#
|
77
54
|
# @return [Fixnum] number of keys that were deleted
|
78
55
|
define_method(:purge_unique_queued_redis_keys) do
|
79
56
|
# unique_at_runtime_key_namespace may or may not ignore the queue passed in, depending on config.
|
80
|
-
key_match = "#{
|
57
|
+
key_match = "#{unique_in_queue_key_namespace(instance_variable_get(:@queue))}:#{unique_in_queue_redis_key_prefix}:*"
|
81
58
|
keys = Resque.redis.keys(key_match)
|
82
|
-
Resque::UniqueByArity.
|
59
|
+
Resque::UniqueByArity.log("#{Resque::UniqueByArity::PLUGIN_TAG}#{Resque::UniqueInQueue::PLUGIN_TAG} #{ColorizedString['Purging'].red} #{keys.length} keys from #{ColorizedString[key_match].red}")
|
83
60
|
Resque.redis.del keys unless keys.empty?
|
84
61
|
end
|
62
|
+
|
85
63
|
if configuration.unique_in_queue
|
86
64
|
# @return [String] the Redis namespace of the key used to enforce uniqueness (at queue-time)
|
87
|
-
define_method(:
|
88
|
-
"#{
|
65
|
+
define_method(:unique_in_queue_key_namespace) do |queue = nil|
|
66
|
+
"#{unique_in_queue_key_base}:queue:#{queue}:job"
|
89
67
|
end
|
90
68
|
elsif configuration.unique_across_queues
|
91
69
|
# @return [String] the Redis namespace of the key used to enforce uniqueness (at queue-time)
|
92
|
-
define_method(:
|
93
|
-
"#{
|
70
|
+
define_method(:unique_in_queue_key_namespace) do |_queue = nil|
|
71
|
+
"#{unique_in_queue_key_base}:across_queues:job"
|
94
72
|
end
|
95
73
|
end
|
96
74
|
end
|
@@ -101,27 +79,26 @@ module Resque
|
|
101
79
|
if configuration.unique_at_runtime
|
102
80
|
# @return [String] the Redis namespace of the key used to enforce uniqueness (at runtime)
|
103
81
|
define_method(:runtime_key_namespace) do
|
104
|
-
"#{
|
82
|
+
"#{unique_at_runtime_key_base}:#{self}"
|
105
83
|
end
|
106
84
|
# Returns a string, used by Resque::Plugins::UniqueAtRuntime, that will be used as the redis key
|
107
|
-
# The versions of redis_key from
|
108
|
-
# So
|
109
|
-
#
|
110
|
-
#
|
111
|
-
#
|
112
|
-
# and we handle the arity option there
|
85
|
+
# The versions of redis_key from resque-solo and resque-lonely_job are incompatible.
|
86
|
+
# So I forked and namespaced the methods according to the gem handling the key.
|
87
|
+
# Now I can override it, and fix the params to be compatible.
|
88
|
+
# Does not need any customization for arity, because ultimately it
|
89
|
+
# still funnels down to redis_key, and we handle the arity option there
|
113
90
|
# @return [String] the key used to enforce loneliness (uniqueness at runtime)
|
114
91
|
define_method(:unique_at_runtime_redis_key) do |*args|
|
115
92
|
unique_hash, args_for_uniqueness = redis_unique_hash('class' => to_s, 'args' => args)
|
116
93
|
key = "#{runtime_key_namespace}:#{unique_hash}"
|
117
|
-
Resque::UniqueByArity.
|
94
|
+
Resque::UniqueByArity.debug("#{Resque::UniqueAtRuntime::PLUGIN_TAG} #{self}.unique_at_runtime_redis_key for #{args_for_uniqueness} is: #{ColorizedString[key].yellow}")
|
118
95
|
key
|
119
96
|
end
|
120
97
|
# @return [Fixnum] number of keys that were deleted
|
121
98
|
define_method(:purge_unique_at_runtime_redis_keys) do
|
122
99
|
key_match = "#{runtime_key_namespace}:*"
|
123
100
|
keys = Resque.redis.keys(key_match)
|
124
|
-
Resque::UniqueByArity.
|
101
|
+
Resque::UniqueByArity.log("#{Resque::UniqueByArity::PLUGIN_TAG}#{Resque::UniqueAtRuntime::PLUGIN_TAG} #{ColorizedString['Purging'].red} #{keys.length} keys from #{ColorizedString[key_match].red}")
|
125
102
|
Resque.redis.del keys unless keys.empty?
|
126
103
|
end
|
127
104
|
end
|
@@ -0,0 +1,53 @@
|
|
1
|
+
# Usage:
|
2
|
+
#
|
3
|
+
# class MyJob
|
4
|
+
# def self.perform(arg1, arg2)
|
5
|
+
# end
|
6
|
+
# include Resque::Plugins::UniqueByArity.new(
|
7
|
+
# arity_for_uniqueness: 1,
|
8
|
+
# arity_validation: :warning, # or nil, false, or :error
|
9
|
+
# unique_at_runtime: true,
|
10
|
+
# unique_in_queue: true
|
11
|
+
# )
|
12
|
+
# end
|
13
|
+
#
|
14
|
+
# NOTE: DO NOT include this module directly.
|
15
|
+
# Use the Resque::Plugins::UniqueByArity approach as above.
|
16
|
+
# This module is ultimately extended into the job class.
|
17
|
+
module Resque
|
18
|
+
module UniqueByArity
|
19
|
+
module UniqueJob
|
20
|
+
PLUGIN_TAG = (ColorizedString['[R-UBA] '].green).freeze
|
21
|
+
|
22
|
+
def uniq_log(message, config_proxy = nil)
|
23
|
+
config_proxy ||= uniq_config
|
24
|
+
config_proxy.logger&.send(config_proxy.log_level, message) if config_proxy.logger
|
25
|
+
end
|
26
|
+
|
27
|
+
def uniq_debug(message, config_proxy = nil)
|
28
|
+
config_proxy ||= uniq_config
|
29
|
+
config_proxy.logger&.debug("#{Resque::UniqueByArity::PLUGIN_TAG}#{message}") if config_proxy.debug_mode
|
30
|
+
end
|
31
|
+
|
32
|
+
# For per-class config with a block
|
33
|
+
def uniqueness_configure
|
34
|
+
@uniqueness_configuration ||= Configuration.new
|
35
|
+
yield(@uniqueness_configuration)
|
36
|
+
end
|
37
|
+
|
38
|
+
#### CONFIG ####
|
39
|
+
class << self
|
40
|
+
attr_accessor :uniqueness_configuration
|
41
|
+
end
|
42
|
+
self.uniqueness_configuration = Configuration.new # setup defaults
|
43
|
+
|
44
|
+
def uniqueness_config_reset(config = Configuration.new)
|
45
|
+
@uniqueness_configuration = config
|
46
|
+
end
|
47
|
+
|
48
|
+
def uniq_config
|
49
|
+
@uniqueness_configuration
|
50
|
+
end
|
51
|
+
end
|
52
|
+
end
|
53
|
+
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: resque-unique_by_arity
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 3.0.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Peter H. Boling
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2018-11-
|
11
|
+
date: 2018-11-15 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: colorize
|
@@ -216,7 +216,6 @@ extra_rdoc_files: []
|
|
216
216
|
files:
|
217
217
|
- ".gitignore"
|
218
218
|
- ".rspec"
|
219
|
-
- ".rspec_status"
|
220
219
|
- ".rubocop.yml"
|
221
220
|
- ".rubocop_todo.yml"
|
222
221
|
- ".ruby-version"
|
@@ -227,10 +226,13 @@ files:
|
|
227
226
|
- Rakefile
|
228
227
|
- bin/console
|
229
228
|
- bin/setup
|
229
|
+
- lib/resque-unique_by_arity.rb
|
230
230
|
- lib/resque/plugins/unique_by_arity.rb
|
231
231
|
- lib/resque/unique_by_arity.rb
|
232
232
|
- lib/resque/unique_by_arity/configuration.rb
|
233
|
+
- lib/resque/unique_by_arity/global_configuration.rb
|
233
234
|
- lib/resque/unique_by_arity/modulizer.rb
|
235
|
+
- lib/resque/unique_by_arity/unique_job.rb
|
234
236
|
- lib/resque/unique_by_arity/validation.rb
|
235
237
|
- lib/resque/unique_by_arity/version.rb
|
236
238
|
- resque-unique_by_arity.gemspec
|
data/.rspec_status
DELETED
@@ -1,55 +0,0 @@
|
|
1
|
-
example_id | status | run_time |
|
2
|
-
----------------------------------------------------------------- | ------ | --------------- |
|
3
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:1] | passed | 0.00313 seconds |
|
4
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:2:1] | passed | 0.00099 seconds |
|
5
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:3:1] | passed | 0.00038 seconds |
|
6
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:4:1] | passed | 0.00016 seconds |
|
7
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:5:1] | passed | 0.00017 seconds |
|
8
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:6:1] | passed | 0.00018 seconds |
|
9
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:7:1] | passed | 0.0002 seconds |
|
10
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:8:1] | passed | 0.00016 seconds |
|
11
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:9:1] | passed | 0.00014 seconds |
|
12
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:10:1] | passed | 0.00014 seconds |
|
13
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:11:1] | passed | 0.00017 seconds |
|
14
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:12:1] | passed | 0.00015 seconds |
|
15
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:13:1] | passed | 0.00017 seconds |
|
16
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:14:1] | passed | 0.00096 seconds |
|
17
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:14:2:1] | passed | 0.00014 seconds |
|
18
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:14:3:1] | passed | 0.00013 seconds |
|
19
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:2:1] | passed | 0.01014 seconds |
|
20
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:3:1] | passed | 0.00031 seconds |
|
21
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:3:2] | passed | 0.00024 seconds |
|
22
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:4:1] | passed | 0.00017 seconds |
|
23
|
-
./spec/resque/unique_by_arity_spec.rb[1:1] | passed | 0.00006 seconds |
|
24
|
-
./spec/resque/unique_by_arity_spec.rb[1:2:1:1] | passed | 0.00094 seconds |
|
25
|
-
./spec/resque/unique_by_arity_spec.rb[1:3:1] | passed | 0.00062 seconds |
|
26
|
-
./spec/resque/unique_by_arity_spec.rb[1:4:1:1] | passed | 0.00022 seconds |
|
27
|
-
./spec/resque/unique_by_arity_spec.rb[1:5:1:1] | passed | 0.00086 seconds |
|
28
|
-
./spec/resque/unique_by_arity_spec.rb[1:5:2:1] | passed | 0.00026 seconds |
|
29
|
-
./spec/resque/unique_by_arity_spec.rb[1:5:3:1] | passed | 0.00027 seconds |
|
30
|
-
./spec/resque/unique_by_arity_spec.rb[1:6:1] | passed | 0.00047 seconds |
|
31
|
-
./spec/resque/unique_by_arity_spec.rb[1:7:1] | passed | 0.00021 seconds |
|
32
|
-
./spec/resque/unique_by_arity_spec.rb[1:7:2:1] | passed | 0.00057 seconds |
|
33
|
-
./spec/resque/unique_by_arity_spec.rb[1:7:3:1] | passed | 0.00062 seconds |
|
34
|
-
./spec/resque/unique_by_arity_spec.rb[1:7:4:1] | passed | 0.00021 seconds |
|
35
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:1:1:1:1] | passed | 0.00028 seconds |
|
36
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:1:2:1:1] | passed | 0.00025 seconds |
|
37
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:1:3:1:1] | passed | 0.00061 seconds |
|
38
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:2:1:1:1] | passed | 0.00022 seconds |
|
39
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:2:2:1:1] | passed | 0.00029 seconds |
|
40
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:2:3:1:1] | passed | 0.00057 seconds |
|
41
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:3:1:1:1] | passed | 0.00076 seconds |
|
42
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:3:2:1:1] | passed | 0.00075 seconds |
|
43
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:3:3:1:1] | passed | 0.00062 seconds |
|
44
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:4:1:1:1] | passed | 0.0004 seconds |
|
45
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:4:2:1:1] | passed | 0.00032 seconds |
|
46
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:4:3:1:1] | passed | 0.00042 seconds |
|
47
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:5:1:1:1] | passed | 0.00023 seconds |
|
48
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:5:2:1:1] | passed | 0.00029 seconds |
|
49
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:5:3:1] | passed | 0.0002 seconds |
|
50
|
-
./spec/resque/unique_by_arity_spec.rb[1:9:1] | passed | 0.0025 seconds |
|
51
|
-
./spec/resque/unique_by_arity_spec.rb[1:10:1:1] | passed | 0.00012 seconds |
|
52
|
-
./spec/resque/unique_by_arity_spec.rb[1:11:1:1] | passed | 0.00029 seconds |
|
53
|
-
./spec/resque/unique_by_arity_spec.rb[1:11:2:1:1] | passed | 0.00025 seconds |
|
54
|
-
./spec/resque/unique_by_arity_spec.rb[1:11:2:2:1] | passed | 0.0006 seconds |
|
55
|
-
./spec/resque/unique_by_arity_spec.rb[1:11:2:3:1] | passed | 0.0005 seconds |
|