resque-unique_by_arity 2.0.2 → 3.0.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.gitignore +2 -0
- data/Gemfile +2 -2
- data/README.md +56 -9
- data/lib/resque-unique_by_arity.rb +17 -0
- data/lib/resque/plugins/unique_by_arity.rb +35 -4
- data/lib/resque/unique_by_arity.rb +18 -116
- data/lib/resque/unique_by_arity/configuration.rb +44 -26
- data/lib/resque/unique_by_arity/global_configuration.rb +52 -0
- data/lib/resque/unique_by_arity/modulizer.rb +20 -43
- data/lib/resque/unique_by_arity/unique_job.rb +53 -0
- data/lib/resque/unique_by_arity/version.rb +1 -1
- metadata +5 -3
- data/.rspec_status +0 -55
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: bfe4371869efe4deb3aed04f46d2417ab32c82e4fadc7954d6d79a0e9cbbe8d2
|
4
|
+
data.tar.gz: 7b8392ab635f8acc02c6e0d9fa09fe79fd287099ede2f75fd654c43661ab161d
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 8e93042522fa854d08126b163847f5fb670321da013f1e7c20a179311aaf7e0c621855d8998cb465e96df38057a7aaa45357cba36caea1ebc035972a65f17727
|
7
|
+
data.tar.gz: 8a9033f210f024ef2cf99805a97e87137b585dbc6ad82ba58c410b2b6dd7d6021cf2f33af84647a85f4aa9c182ef76830674afa81628c5bbd046625efcb3558e
|
data/.gitignore
CHANGED
data/Gemfile
CHANGED
@@ -2,8 +2,8 @@ source 'https://rubygems.org'
|
|
2
2
|
|
3
3
|
git_source(:github) { |repo_name| "https://github.com/#{repo_name}" }
|
4
4
|
|
5
|
-
gem 'resque-unique_in_queue'
|
6
|
-
gem 'resque-unique_at_runtime'
|
5
|
+
gem 'resque-unique_in_queue', path: '/Users/pboling/Documents/code/intricately/resque-unique_in_queue'
|
6
|
+
gem 'resque-unique_at_runtime', path: '/Users/pboling/Documents/code/intricately/resque-unique_at_runtime'
|
7
7
|
|
8
8
|
group :test do
|
9
9
|
unless ENV['TRAVIS']
|
data/README.md
CHANGED
@@ -1,11 +1,18 @@
|
|
1
1
|
# Resque::UniqueByArity
|
2
2
|
|
3
|
-
Because some jobs have parameters that you do not want to consider for
|
3
|
+
Because some jobs have parameters that you do not want to consider for
|
4
|
+
determination of uniqueness.
|
4
5
|
|
5
6
|
NOTE:
|
6
7
|
|
7
|
-
|
8
|
-
|
8
|
+
I rewrote, and renamed, both `resque_solo` and `resque-lonely_job`, becuase they
|
9
|
+
can't be used together. Why? Their `redis_key` methods directly conflict,
|
10
|
+
among other more subtle issues.
|
11
|
+
|
12
|
+
This gem requires use of my rewritten gems for uniqueness enforcement:
|
13
|
+
|
14
|
+
- [`resque-unique_at_runtime`](https://github.com/pboling/resque-unique_at_runtime)
|
15
|
+
- [`resque-unique_in_queue`](https://github.com/pboling/resque-unique_in_queue)
|
9
16
|
|
10
17
|
| Project | Resque::UniqueByArity |
|
11
18
|
|------------------------ | ----------------------- |
|
@@ -24,11 +31,13 @@ Why? `resque-lonely_job` and `resque_solo` can't be used together, because their
|
|
24
31
|
|
25
32
|
## Important Note
|
26
33
|
|
27
|
-
See `lib/resque/unique_by_arity/configuration.rb` for all config options. Only
|
34
|
+
See `lib/resque/unique_by_arity/configuration.rb` for all config options. Only
|
35
|
+
a smattering of what is available is documented in this README.
|
28
36
|
|
29
37
|
## Most Important Note
|
30
38
|
|
31
|
-
You must configure this gem *after* you define the perform class method in your
|
39
|
+
You must configure this gem *after* you define the perform class method in your
|
40
|
+
job or an error will be raised thanks to `perform` not having been defined yet.
|
32
41
|
|
33
42
|
Example:
|
34
43
|
|
@@ -65,9 +74,44 @@ Or install it yourself as:
|
|
65
74
|
|
66
75
|
## Usage
|
67
76
|
|
77
|
+
This gem will take care to set the class instance variables (similar to the
|
78
|
+
familiar `@queue` class instance variable) that are utilized by
|
79
|
+
`resque-unique_in_queue` and `resque-unique_at_runtime` (default values shown):
|
80
|
+
|
81
|
+
```ruby
|
82
|
+
# For resque-unique_at_runtime
|
83
|
+
@runtime_lock_timeout = 60 * 60 * 24 * 5
|
84
|
+
@runtime_requeue_interval = 1
|
85
|
+
@unique_at_runtime_key_base = 'r-uar'.freeze
|
86
|
+
|
87
|
+
# For resque-unique_in_queue
|
88
|
+
@lock_after_execution_period = 0
|
89
|
+
@ttl = -1
|
90
|
+
@unique_in_queue_key_base = 'r-uiq'.freeze
|
91
|
+
```
|
92
|
+
|
93
|
+
All you need to do is configure this gem accordingly:
|
94
|
+
```ruby
|
95
|
+
include Resque::Plugins::UniqueByArity.new(
|
96
|
+
arity_for_uniqueness: 3,
|
97
|
+
unique_at_runtime: true,
|
98
|
+
unique_in_queue: true,
|
99
|
+
# No need to do the following if keeping default values
|
100
|
+
runtime_lock_timeout: 60 * 60 * 24 * 5,
|
101
|
+
runtime_requeue_interval: 1,
|
102
|
+
unique_at_runtime_key_base: 'r-uar'.freeze,
|
103
|
+
lock_after_execution_period: 0,
|
104
|
+
ttl: 0,
|
105
|
+
unique_in_queue_key_base: 'r-uiq'.freeze
|
106
|
+
)
|
107
|
+
```
|
108
|
+
|
68
109
|
### Arity For Uniqueness
|
69
110
|
|
70
|
-
Some jobs have parameters that you do not want to consider for determination of
|
111
|
+
Some jobs have parameters that you do not want to consider for determination of
|
112
|
+
uniqueness. Resque jobs should use simple parameters, **not named parameters**,
|
113
|
+
so you can just specify the number of parameters, counting from the left, you
|
114
|
+
want to be considered for uniqueness.
|
71
115
|
|
72
116
|
```ruby
|
73
117
|
class MyJob
|
@@ -105,7 +149,8 @@ end
|
|
105
149
|
|
106
150
|
### Lock After Execution
|
107
151
|
|
108
|
-
Give the job a break after it finishes running, and don't allow another of the
|
152
|
+
Give the job a break after it finishes running, and don't allow another of the
|
153
|
+
same, with matching args @ configured arity, to start within X seconds.
|
109
154
|
|
110
155
|
```ruby
|
111
156
|
class MyJob
|
@@ -122,7 +167,8 @@ end
|
|
122
167
|
|
123
168
|
### Runtime Lock Timeout
|
124
169
|
|
125
|
-
If runtime lock keys get stale, they will expire on their own after some period.
|
170
|
+
If runtime lock keys get stale, they will expire on their own after some period.
|
171
|
+
You can set the expiration period on a per class basis.
|
126
172
|
|
127
173
|
```ruby
|
128
174
|
class MyJob
|
@@ -263,7 +309,8 @@ end
|
|
263
309
|
|
264
310
|
### Debugging
|
265
311
|
|
266
|
-
Run your worker with `RESQUE_DEBUG=true` to see payloads printed before they are
|
312
|
+
Run your worker with `RESQUE_DEBUG=true` to see payloads printed before they are
|
313
|
+
used to determine uniqueness, as well as a lot of other debugging output.
|
267
314
|
|
268
315
|
### Customize Unique Keys Per Job
|
269
316
|
|
@@ -0,0 +1,17 @@
|
|
1
|
+
require 'resque/unique_by_arity/version'
|
2
|
+
|
3
|
+
# External Gems
|
4
|
+
require 'colorized_string'
|
5
|
+
require 'resque'
|
6
|
+
|
7
|
+
# External Resque Plugins
|
8
|
+
require 'resque-unique_in_queue'
|
9
|
+
require 'resque-unique_at_runtime'
|
10
|
+
|
11
|
+
require 'resque/unique_by_arity/configuration'
|
12
|
+
require 'resque/unique_by_arity/global_configuration'
|
13
|
+
require 'resque/unique_by_arity'
|
14
|
+
require 'resque/unique_by_arity/modulizer'
|
15
|
+
require 'resque/unique_by_arity/unique_job'
|
16
|
+
require 'resque/unique_by_arity/validation'
|
17
|
+
require 'resque/plugins/unique_by_arity'
|
@@ -35,17 +35,48 @@ module Resque
|
|
35
35
|
# As a result we can configure per class.
|
36
36
|
@configuration.base_klass_name = base.to_s
|
37
37
|
@configuration.validate
|
38
|
-
base.send(:extend, Resque::UniqueByArity)
|
38
|
+
base.send(:extend, Resque::UniqueByArity::UniqueJob)
|
39
39
|
base.uniqueness_config_reset(@configuration.dup)
|
40
40
|
|
41
|
-
# gem is unique_in_queue, which is a rewrite of resque-loner
|
42
|
-
# see: https://github.com/
|
41
|
+
# gem is resque-unique_in_queue, which is a rewrite of resque-solo / resque-loner
|
42
|
+
# see: https://github.com/pboling/resque-unique_in_queue
|
43
43
|
# defines a redis_key method, which we have to override.
|
44
44
|
base.send(:include, Resque::Plugins::UniqueInQueue) if @configuration.unique_in_queue || @configuration.unique_across_queues
|
45
45
|
|
46
46
|
# gem is resque-unique_at_runtime, which is a rewrite of resque-lonely_job
|
47
47
|
# see: https://github.com/pboling/resque-unique_at_runtime
|
48
|
-
base.send(:
|
48
|
+
base.send(:include, Resque::Plugins::UniqueAtRuntime) if @configuration.unique_at_runtime
|
49
|
+
|
50
|
+
# For resque-unique_at_runtime
|
51
|
+
#
|
52
|
+
if @configuration.runtime_lock_timeout
|
53
|
+
base.instance_variable_set(:@runtime_lock_timeout, @configuration.runtime_lock_timeout)
|
54
|
+
end
|
55
|
+
|
56
|
+
if @configuration.runtime_requeue_interval
|
57
|
+
base.instance_variable_set(:@runtime_requeue_interval, @configuration.runtime_requeue_interval)
|
58
|
+
end
|
59
|
+
|
60
|
+
if @configuration.unique_at_runtime_key_base
|
61
|
+
base.instance_variable_set(:@unique_at_runtime_key_base, @configuration.unique_at_runtime_key_base)
|
62
|
+
end
|
63
|
+
|
64
|
+
# For resque-unique_in_queue
|
65
|
+
#
|
66
|
+
if @configuration.lock_after_execution_period
|
67
|
+
base.instance_variable_set(:@lock_after_execution_period, @configuration.lock_after_execution_period)
|
68
|
+
end
|
69
|
+
|
70
|
+
if @configuration.ttl
|
71
|
+
base.instance_variable_set(:@ttl, @configuration.ttl)
|
72
|
+
end
|
73
|
+
|
74
|
+
# Normally doesn't make sense to override per each class because
|
75
|
+
# it wouldn't be able to determine or enforce uniqueness across queues,
|
76
|
+
# and general cleanup of stray keys would be nearly impossible.
|
77
|
+
if @configuration.unique_in_queue_key_base
|
78
|
+
base.instance_variable_set(:@unique_in_queue_key_base, @configuration.unique_in_queue_key_base)
|
79
|
+
end
|
49
80
|
|
50
81
|
uniqueness_cop_module = Resque::UniqueByArity::Modulizer.to_mod(@configuration)
|
51
82
|
# This will override methods from both plugins above, if configured for both
|
@@ -1,132 +1,34 @@
|
|
1
|
-
require 'resque/unique_by_arity
|
1
|
+
# Just in case some code still does require 'resque/unique_by_arity'
|
2
|
+
require 'resque-unique_by_arity'
|
2
3
|
|
3
|
-
# External Gems
|
4
|
-
require 'colorized_string'
|
5
|
-
require 'resque'
|
6
|
-
|
7
|
-
# External Resque Plugins
|
8
|
-
require 'resque-unique_in_queue'
|
9
|
-
require 'resque-unique_at_runtime'
|
10
|
-
|
11
|
-
require 'resque/plugins/unique_by_arity'
|
12
|
-
require 'resque/unique_by_arity/configuration'
|
13
|
-
require 'resque/unique_by_arity/modulizer'
|
14
|
-
require 'resque/unique_by_arity/validation'
|
15
|
-
|
16
|
-
# Usage:
|
17
|
-
#
|
18
|
-
# class MyJob
|
19
|
-
# def self.perform(arg1, arg2)
|
20
|
-
# end
|
21
|
-
# include Resque::Plugins::UniqueByArity.new(
|
22
|
-
# arity_for_uniqueness: 1,
|
23
|
-
# arity_validation: :warning, # or nil, false, or :error
|
24
|
-
# unique_at_runtime: true,
|
25
|
-
# unique_in_queue: true
|
26
|
-
# )
|
27
|
-
# end
|
28
|
-
#
|
29
|
-
# NOTE: DO NOT include this module directly.
|
30
|
-
# Use the Resque::Plugins::UniqueByArity approach as above.
|
31
|
-
# This module is ultimately extended into the job class.
|
32
4
|
module Resque
|
33
5
|
module UniqueByArity
|
34
6
|
PLUGIN_TAG = (ColorizedString['[R-UBA] '].green).freeze
|
35
7
|
|
36
|
-
def
|
37
|
-
config_proxy ||=
|
38
|
-
config_proxy.
|
39
|
-
end
|
40
|
-
|
41
|
-
def unique_debug(message, config_proxy = nil)
|
42
|
-
config_proxy ||= uniqueness_configuration
|
43
|
-
config_proxy.unique_logger&.debug("#{Resque::UniqueByArity::PLUGIN_TAG}#{message}") if config_proxy.debug_mode
|
8
|
+
def log(message, config_proxy = nil)
|
9
|
+
config_proxy ||= configuration
|
10
|
+
config_proxy.logger&.send(config_proxy.log_level, message) if config_proxy.logger
|
44
11
|
end
|
45
|
-
module_function(:
|
12
|
+
module_function(:log)
|
46
13
|
|
47
|
-
|
48
|
-
|
49
|
-
|
50
|
-
@uniqueness_configuration
|
14
|
+
def debug(message, config_proxy = nil)
|
15
|
+
config_proxy ||= configuration
|
16
|
+
config_proxy.logger&.debug("#{Resque::UniqueByArity::PLUGIN_TAG}#{message}") if config_proxy.debug_mode
|
51
17
|
end
|
18
|
+
module_function(:debug)
|
52
19
|
|
53
|
-
# For
|
54
|
-
def
|
55
|
-
@
|
56
|
-
yield(@uniqueness_configuration)
|
20
|
+
# For config with a block
|
21
|
+
def configure
|
22
|
+
yield(@configuration)
|
57
23
|
end
|
58
24
|
|
59
25
|
#### CONFIG ####
|
26
|
+
# Access globally configured settings:
|
27
|
+
# >> Resque::UniqueByArity.configuration.logger
|
28
|
+
# => the Logger instance
|
60
29
|
class << self
|
61
|
-
attr_accessor :
|
62
|
-
end
|
63
|
-
def uniqueness_config_reset(config = Configuration.new)
|
64
|
-
@uniqueness_configuration = config
|
65
|
-
end
|
66
|
-
|
67
|
-
def uniqueness_log_level
|
68
|
-
@uniqueness_configuration.log_level
|
69
|
-
end
|
70
|
-
|
71
|
-
def uniqueness_log_level=(log_level)
|
72
|
-
@uniqueness_configuration.log_level = log_level
|
73
|
-
end
|
74
|
-
|
75
|
-
def uniqueness_arity_for_uniqueness
|
76
|
-
@uniqueness_configuration.arity_for_uniqueness
|
77
|
-
end
|
78
|
-
|
79
|
-
def uniqueness_arity_for_uniqueness=(arity_for_uniqueness)
|
80
|
-
@uniqueness_configuration.arity_for_uniqueness = arity_for_uniqueness
|
81
|
-
end
|
82
|
-
|
83
|
-
def uniqueness_arity_validation
|
84
|
-
@uniqueness_configuration.arity_validation
|
85
|
-
end
|
86
|
-
|
87
|
-
def uniqueness_arity_validation=(arity_validation)
|
88
|
-
@uniqueness_configuration.arity_validation = arity_validation
|
89
|
-
end
|
90
|
-
|
91
|
-
def uniqueness_lock_after_execution_period
|
92
|
-
@uniqueness_configuration.lock_after_execution_period
|
93
|
-
end
|
94
|
-
|
95
|
-
def uniqueness_lock_after_execution_period=(lock_after_execution_period)
|
96
|
-
@uniqueness_configuration.lock_after_execution_period = lock_after_execution_period
|
97
|
-
end
|
98
|
-
|
99
|
-
def uniqueness_runtime_lock_timeout
|
100
|
-
@uniqueness_configuration.runtime_lock_timeout
|
101
|
-
end
|
102
|
-
|
103
|
-
def uniqueness_runtime_lock_timeout=(runtime_lock_timeout)
|
104
|
-
@uniqueness_configuration.runtime_lock_timeout = runtime_lock_timeout
|
105
|
-
end
|
106
|
-
|
107
|
-
def uniqueness_unique_at_runtime
|
108
|
-
@uniqueness_configuration.unique_at_runtime
|
109
|
-
end
|
110
|
-
|
111
|
-
def uniqueness_unique_at_runtime=(unique_at_runtime)
|
112
|
-
@uniqueness_configuration.unique_at_runtime = unique_at_runtime
|
113
|
-
end
|
114
|
-
|
115
|
-
def uniqueness_unique_in_queue
|
116
|
-
@uniqueness_configuration.unique_in_queue
|
117
|
-
end
|
118
|
-
|
119
|
-
def uniqueness_unique_in_queue=(unique_in_queue)
|
120
|
-
@uniqueness_configuration.unique_in_queue = unique_in_queue
|
121
|
-
end
|
122
|
-
|
123
|
-
def uniqueness_unique_across_queues
|
124
|
-
@uniqueness_configuration.unique_across_queues
|
125
|
-
end
|
126
|
-
|
127
|
-
def uniqueness_unique_across_queues=(unique_across_queues)
|
128
|
-
@uniqueness_configuration.unique_across_queues = unique_across_queues
|
30
|
+
attr_accessor :configuration
|
129
31
|
end
|
130
|
-
self.
|
32
|
+
self.configuration = GlobalConfiguration.instance # setup defaults
|
131
33
|
end
|
132
34
|
end
|
@@ -1,11 +1,13 @@
|
|
1
1
|
require 'logger'
|
2
2
|
module Resque
|
3
3
|
module UniqueByArity
|
4
|
+
# This class is for configurations that are per job class, *not* app-wide.
|
5
|
+
# Each setting will default to the global config values.
|
4
6
|
class Configuration
|
5
7
|
VALID_ARITY_VALIDATION_LEVELS = [:warning, :error, :skip, nil, false].freeze
|
6
8
|
SKIPPED_ARITY_VALIDATION_LEVELS = [:skip, nil, false].freeze
|
7
|
-
|
8
|
-
|
9
|
+
DEFAULT_LOG_LEVEL = :debug
|
10
|
+
|
9
11
|
attr_accessor :logger
|
10
12
|
attr_accessor :log_level
|
11
13
|
attr_accessor :arity_for_uniqueness
|
@@ -18,27 +20,35 @@ module Resque
|
|
18
20
|
attr_accessor :unique_in_queue
|
19
21
|
attr_accessor :unique_in_queue_key_base
|
20
22
|
attr_accessor :unique_across_queues
|
23
|
+
attr_accessor :ttl
|
21
24
|
attr_accessor :base_klass_name
|
22
25
|
attr_accessor :debug_mode
|
26
|
+
|
23
27
|
def initialize(**options)
|
24
|
-
@logger = options.key?(:logger) ? options[:logger] : Logger.new(STDOUT)
|
25
|
-
@log_level = options.key?(:log_level) ? options[:log_level] : :debug
|
26
|
-
@arity_for_uniqueness = options.key?(:arity_for_uniqueness) ? options[:arity_for_uniqueness] : 1
|
27
|
-
@arity_validation = options.key?(:arity_validation) ? options[:arity_validation] : :warning
|
28
|
+
@logger = options.key?(:logger) ? options[:logger] : defcon(:logger) || Logger.new(STDOUT)
|
29
|
+
@log_level = options.key?(:log_level) ? options[:log_level] : defcon(:log_level) || :debug
|
30
|
+
@arity_for_uniqueness = options.key?(:arity_for_uniqueness) ? options[:arity_for_uniqueness] : defcon(:arity_for_uniqueness) || 1
|
31
|
+
@arity_validation = options.key?(:arity_validation) ? options[:arity_validation] : defcon(:arity_validation) || :warning
|
28
32
|
raise ArgumentError, "Resque::Plugins::UniqueByArity.new requires arity_validation values of #{arity_validation.inspect}, or a class inheriting from Exception, but the value is #{@arity_validation} (#{@arity_validation.class})" unless VALID_ARITY_VALIDATION_LEVELS.include?(@arity_validation) || !@arity_validation.respond_to?(:ancestors) || @arity_validation.ancestors.include?(Exception)
|
29
33
|
|
30
|
-
@
|
31
|
-
@
|
32
|
-
@
|
33
|
-
@
|
34
|
-
@
|
35
|
-
@
|
36
|
-
@
|
37
|
-
@
|
38
|
-
|
34
|
+
@ttl = options.key?(:ttl) ? options[:ttl] : defcon(:ttl) || nil
|
35
|
+
@lock_after_execution_period = options.key?(:lock_after_execution_period) ? options[:lock_after_execution_period] : defcon(:lock_after_execution_period) || nil
|
36
|
+
@runtime_lock_timeout = options.key?(:runtime_lock_timeout) ? options[:runtime_lock_timeout] : defcon(:runtime_lock_timeout) || nil
|
37
|
+
@runtime_requeue_interval = options.key?(:runtime_requeue_interval) ? options[:runtime_requeue_interval] : defcon(:runtime_requeue_interval) || nil
|
38
|
+
@unique_at_runtime = options.key?(:unique_at_runtime) ? options[:unique_at_runtime] : defcon(:unique_at_runtime) || false
|
39
|
+
@unique_at_runtime_key_base = options.key?(:unique_at_runtime_key_base) ? options[:unique_at_runtime_key_base] : defcon(:unique_at_runtime_key_base) || nil
|
40
|
+
@unique_in_queue_key_base = options.key?(:unique_in_queue_key_base) ? options[:unique_in_queue_key_base] : defcon(:unique_in_queue_key_base) || nil
|
41
|
+
@unique_in_queue = options.key?(:unique_in_queue) ? options[:unique_in_queue] : defcon(:unique_in_queue) || false
|
42
|
+
@unique_across_queues = options.key?(:unique_across_queues) ? options[:unique_across_queues] : defcon(:unique_across_queues) || false
|
43
|
+
# Can't be both unique in queue and unique across queues, since they
|
44
|
+
# must necessarily use different locking key structures, and therefore
|
45
|
+
# can't both be active.
|
39
46
|
raise ArgumentError, "Resque::Plugins::UniqueByArity.new requires either one or none of @unique_across_queues and @unique_in_queue to be true. Having both set to true is non-sensical." if @unique_in_queue && @unique_across_queues
|
40
|
-
|
41
|
-
@debug_mode
|
47
|
+
@debug_mode = !!(options.key?(:debug_mode) ? options[:debug_mode] : defcon(:debug_mode))
|
48
|
+
if @debug_mode
|
49
|
+
# Make sure there is a logger when in debug_mode
|
50
|
+
@logger ||= Logger.new(STDOUT)
|
51
|
+
end
|
42
52
|
end
|
43
53
|
|
44
54
|
def validate
|
@@ -52,16 +62,8 @@ module Resque
|
|
52
62
|
end
|
53
63
|
end
|
54
64
|
|
55
|
-
def unique_logger
|
56
|
-
logger
|
57
|
-
end
|
58
|
-
|
59
|
-
def unique_log_level
|
60
|
-
log_level
|
61
|
-
end
|
62
|
-
|
63
65
|
def log(msg)
|
64
|
-
Resque::UniqueByArity.
|
66
|
+
Resque::UniqueByArity.log(msg, self)
|
65
67
|
end
|
66
68
|
|
67
69
|
def to_hash
|
@@ -74,6 +76,7 @@ module Resque
|
|
74
76
|
debug_mode: debug_mode,
|
75
77
|
lock_after_execution_period: lock_after_execution_period,
|
76
78
|
runtime_lock_timeout: runtime_lock_timeout,
|
79
|
+
ttl: ttl,
|
77
80
|
unique_at_runtime: unique_at_runtime,
|
78
81
|
unique_in_queue: unique_in_queue,
|
79
82
|
unique_across_queues: unique_across_queues
|
@@ -121,6 +124,21 @@ module Resque
|
|
121
124
|
end
|
122
125
|
end
|
123
126
|
end
|
127
|
+
|
128
|
+
def defcon(sym)
|
129
|
+
Resque::UniqueByArity.configuration.send(sym)
|
130
|
+
end
|
131
|
+
|
132
|
+
def debug_mode=(val)
|
133
|
+
@debug_mode = !!val
|
134
|
+
end
|
135
|
+
|
136
|
+
private
|
137
|
+
|
138
|
+
def debug_mode_from_env
|
139
|
+
env_debug = ENV['RESQUE_DEBUG']
|
140
|
+
@debug_mode = !!(env_debug == 'true' || (env_debug.is_a?(String) && env_debug.match?(/queue/)))
|
141
|
+
end
|
124
142
|
end
|
125
143
|
end
|
126
144
|
end
|
@@ -0,0 +1,52 @@
|
|
1
|
+
require 'logger'
|
2
|
+
module Resque
|
3
|
+
module UniqueByArity
|
4
|
+
# This class is for configurations that are app-wide, *not per job class.
|
5
|
+
# For this reason it is a Singleton.
|
6
|
+
# Will be used as the default settings for the per-job configs.
|
7
|
+
class GlobalConfiguration < Configuration
|
8
|
+
DEFAULT_LOG_LEVEL = :debug
|
9
|
+
DEFAULT_AT_RUNTIME_KEY_BASE = 'r-uar'.freeze
|
10
|
+
DEFAULT_IN_QUEUE_KEY_BASE = 'r-uiq'.freeze
|
11
|
+
|
12
|
+
# For resque-unique_at_runtime
|
13
|
+
DEFAULT_LOCK_TIMEOUT = 60 * 60 * 24 * 5
|
14
|
+
DEFAULT_REQUEUE_INTERVAL = 1
|
15
|
+
DEFAULT_UNIQUE_AT_RUNTIME_KEY_BASE = 'r-uar'.freeze
|
16
|
+
|
17
|
+
# For resque-unique_in_queue
|
18
|
+
DEFAULT_LOCK_AFTER_EXECUTION_PERIOD = 0
|
19
|
+
DEFAULT_TTL = -1
|
20
|
+
DEFAULT_UNIQUE_IN_QUEUE_KEY_BASE = 'r-uiq'.freeze
|
21
|
+
|
22
|
+
include Singleton
|
23
|
+
|
24
|
+
def initialize
|
25
|
+
debug_mode_from_env
|
26
|
+
@logger = nil
|
27
|
+
@log_level = DEFAULT_LOG_LEVEL
|
28
|
+
@arity_for_uniqueness = nil
|
29
|
+
@arity_validation = nil
|
30
|
+
@lock_after_execution_period = DEFAULT_LOCK_AFTER_EXECUTION_PERIOD
|
31
|
+
@runtime_lock_timeout = DEFAULT_LOCK_TIMEOUT
|
32
|
+
@runtime_requeue_interval = DEFAULT_REQUEUE_INTERVAL
|
33
|
+
@unique_at_runtime_key_base = DEFAULT_AT_RUNTIME_KEY_BASE
|
34
|
+
@unique_in_queue_key_base = DEFAULT_IN_QUEUE_KEY_BASE
|
35
|
+
@unique_at_runtime = false
|
36
|
+
@unique_at_runtime_key_base = DEFAULT_UNIQUE_AT_RUNTIME_KEY_BASE
|
37
|
+
@unique_in_queue_key_base = DEFAULT_UNIQUE_IN_QUEUE_KEY_BASE
|
38
|
+
@unique_in_queue = false
|
39
|
+
@unique_across_queues = false
|
40
|
+
@ttl = DEFAULT_TTL
|
41
|
+
if @debug_mode
|
42
|
+
# Make sure there is a logger when in debug_mode
|
43
|
+
@logger ||= Logger.new(STDOUT)
|
44
|
+
end
|
45
|
+
end
|
46
|
+
|
47
|
+
def defcon(sym)
|
48
|
+
self.send(sym)
|
49
|
+
end
|
50
|
+
end
|
51
|
+
end
|
52
|
+
end
|
@@ -7,7 +7,7 @@ module Resque
|
|
7
7
|
# @return [Array<String, arguments>] the key base hash used to enforce uniqueness, and the arguments from the payload used to calculate it
|
8
8
|
define_method(:redis_unique_hash) do |payload|
|
9
9
|
payload = Resque.decode(Resque.encode(payload))
|
10
|
-
Resque::UniqueByArity.
|
10
|
+
Resque::UniqueByArity.debug("payload is #{payload.inspect}")
|
11
11
|
job = payload['class']
|
12
12
|
# It seems possible that some jobs may not have an "args" key in the payload.
|
13
13
|
args = payload['args'] || []
|
@@ -28,36 +28,13 @@ module Resque
|
|
28
28
|
end
|
29
29
|
end
|
30
30
|
|
31
|
-
if configuration.lock_after_execution_period
|
32
|
-
instance_variable_set(:@lock_after_execution_period, configuration.lock_after_execution_period)
|
33
|
-
end
|
34
|
-
|
35
|
-
if configuration.runtime_lock_timeout
|
36
|
-
instance_variable_set(:@runtime_lock_timeout, configuration.runtime_lock_timeout)
|
37
|
-
end
|
38
|
-
|
39
|
-
if configuration.runtime_requeue_interval
|
40
|
-
instance_variable_set(:@runtime_requeue_interval, configuration.runtime_requeue_interval)
|
41
|
-
end
|
42
|
-
|
43
|
-
if configuration.unique_at_runtime_key_base
|
44
|
-
instance_variable_set(:@unique_at_runtime_key_base, configuration.unique_at_runtime_key_base)
|
45
|
-
end
|
46
|
-
|
47
|
-
if configuration.unique_in_queue_key_base
|
48
|
-
# Can't be overridden per each class because it wouldn't make sense.
|
49
|
-
# It wouldn't be able to determine or enforce uniqueness across queues,
|
50
|
-
# and general cleanup of stray keys would be nearly impossible.
|
51
|
-
Resque::UniqueInQueue.uniq_config&.unique_in_queue_key_base = configuration.unique_in_queue_key_base
|
52
|
-
end
|
53
|
-
|
54
31
|
if configuration.unique_in_queue || configuration.unique_across_queues
|
55
|
-
### Gem: unique_in_queue
|
32
|
+
### Gem: resque-unique_in_queue
|
56
33
|
### Plugin Name: Resque::Plugins::UniqueJob
|
57
34
|
### Provides: Queue-time uniqueness for a single queue, or across queues
|
58
35
|
#
|
59
36
|
# Returns a string, used by Resque::Plugins::UniqueJob, that will be used as the prefix to the redis key
|
60
|
-
define_method(:
|
37
|
+
define_method(:unique_in_queue_redis_key_prefix) do
|
61
38
|
"unique_job:#{self}"
|
62
39
|
end
|
63
40
|
#
|
@@ -69,28 +46,29 @@ module Resque
|
|
69
46
|
# @return [String] the key used to enforce uniqueness (at queue-time)
|
70
47
|
define_method(:unique_in_queue_redis_key) do |queue, payload|
|
71
48
|
unique_hash, args_for_uniqueness = redis_unique_hash(payload)
|
72
|
-
key = "#{
|
73
|
-
Resque::UniqueByArity.
|
49
|
+
key = "#{unique_in_queue_key_namespace(queue)}:#{unique_in_queue_redis_key_prefix}:#{unique_hash}"
|
50
|
+
Resque::UniqueByArity.debug("#{self}.unique_in_queue_redis_key for #{args_for_uniqueness} is: #{ColorizedString[key].green}")
|
74
51
|
key
|
75
52
|
end
|
76
53
|
#
|
77
54
|
# @return [Fixnum] number of keys that were deleted
|
78
55
|
define_method(:purge_unique_queued_redis_keys) do
|
79
56
|
# unique_at_runtime_key_namespace may or may not ignore the queue passed in, depending on config.
|
80
|
-
key_match = "#{
|
57
|
+
key_match = "#{unique_in_queue_key_namespace(instance_variable_get(:@queue))}:#{unique_in_queue_redis_key_prefix}:*"
|
81
58
|
keys = Resque.redis.keys(key_match)
|
82
|
-
Resque::UniqueByArity.
|
59
|
+
Resque::UniqueByArity.log("#{Resque::UniqueByArity::PLUGIN_TAG}#{Resque::UniqueInQueue::PLUGIN_TAG} #{ColorizedString['Purging'].red} #{keys.length} keys from #{ColorizedString[key_match].red}")
|
83
60
|
Resque.redis.del keys unless keys.empty?
|
84
61
|
end
|
62
|
+
|
85
63
|
if configuration.unique_in_queue
|
86
64
|
# @return [String] the Redis namespace of the key used to enforce uniqueness (at queue-time)
|
87
|
-
define_method(:
|
88
|
-
"#{
|
65
|
+
define_method(:unique_in_queue_key_namespace) do |queue = nil|
|
66
|
+
"#{unique_in_queue_key_base}:queue:#{queue}:job"
|
89
67
|
end
|
90
68
|
elsif configuration.unique_across_queues
|
91
69
|
# @return [String] the Redis namespace of the key used to enforce uniqueness (at queue-time)
|
92
|
-
define_method(:
|
93
|
-
"#{
|
70
|
+
define_method(:unique_in_queue_key_namespace) do |_queue = nil|
|
71
|
+
"#{unique_in_queue_key_base}:across_queues:job"
|
94
72
|
end
|
95
73
|
end
|
96
74
|
end
|
@@ -101,27 +79,26 @@ module Resque
|
|
101
79
|
if configuration.unique_at_runtime
|
102
80
|
# @return [String] the Redis namespace of the key used to enforce uniqueness (at runtime)
|
103
81
|
define_method(:runtime_key_namespace) do
|
104
|
-
"#{
|
82
|
+
"#{unique_at_runtime_key_base}:#{self}"
|
105
83
|
end
|
106
84
|
# Returns a string, used by Resque::Plugins::UniqueAtRuntime, that will be used as the redis key
|
107
|
-
# The versions of redis_key from
|
108
|
-
# So
|
109
|
-
#
|
110
|
-
#
|
111
|
-
#
|
112
|
-
# and we handle the arity option there
|
85
|
+
# The versions of redis_key from resque-solo and resque-lonely_job are incompatible.
|
86
|
+
# So I forked and namespaced the methods according to the gem handling the key.
|
87
|
+
# Now I can override it, and fix the params to be compatible.
|
88
|
+
# Does not need any customization for arity, because ultimately it
|
89
|
+
# still funnels down to redis_key, and we handle the arity option there
|
113
90
|
# @return [String] the key used to enforce loneliness (uniqueness at runtime)
|
114
91
|
define_method(:unique_at_runtime_redis_key) do |*args|
|
115
92
|
unique_hash, args_for_uniqueness = redis_unique_hash('class' => to_s, 'args' => args)
|
116
93
|
key = "#{runtime_key_namespace}:#{unique_hash}"
|
117
|
-
Resque::UniqueByArity.
|
94
|
+
Resque::UniqueByArity.debug("#{Resque::UniqueAtRuntime::PLUGIN_TAG} #{self}.unique_at_runtime_redis_key for #{args_for_uniqueness} is: #{ColorizedString[key].yellow}")
|
118
95
|
key
|
119
96
|
end
|
120
97
|
# @return [Fixnum] number of keys that were deleted
|
121
98
|
define_method(:purge_unique_at_runtime_redis_keys) do
|
122
99
|
key_match = "#{runtime_key_namespace}:*"
|
123
100
|
keys = Resque.redis.keys(key_match)
|
124
|
-
Resque::UniqueByArity.
|
101
|
+
Resque::UniqueByArity.log("#{Resque::UniqueByArity::PLUGIN_TAG}#{Resque::UniqueAtRuntime::PLUGIN_TAG} #{ColorizedString['Purging'].red} #{keys.length} keys from #{ColorizedString[key_match].red}")
|
125
102
|
Resque.redis.del keys unless keys.empty?
|
126
103
|
end
|
127
104
|
end
|
@@ -0,0 +1,53 @@
|
|
1
|
+
# Usage:
|
2
|
+
#
|
3
|
+
# class MyJob
|
4
|
+
# def self.perform(arg1, arg2)
|
5
|
+
# end
|
6
|
+
# include Resque::Plugins::UniqueByArity.new(
|
7
|
+
# arity_for_uniqueness: 1,
|
8
|
+
# arity_validation: :warning, # or nil, false, or :error
|
9
|
+
# unique_at_runtime: true,
|
10
|
+
# unique_in_queue: true
|
11
|
+
# )
|
12
|
+
# end
|
13
|
+
#
|
14
|
+
# NOTE: DO NOT include this module directly.
|
15
|
+
# Use the Resque::Plugins::UniqueByArity approach as above.
|
16
|
+
# This module is ultimately extended into the job class.
|
17
|
+
module Resque
|
18
|
+
module UniqueByArity
|
19
|
+
module UniqueJob
|
20
|
+
PLUGIN_TAG = (ColorizedString['[R-UBA] '].green).freeze
|
21
|
+
|
22
|
+
def uniq_log(message, config_proxy = nil)
|
23
|
+
config_proxy ||= uniq_config
|
24
|
+
config_proxy.logger&.send(config_proxy.log_level, message) if config_proxy.logger
|
25
|
+
end
|
26
|
+
|
27
|
+
def uniq_debug(message, config_proxy = nil)
|
28
|
+
config_proxy ||= uniq_config
|
29
|
+
config_proxy.logger&.debug("#{Resque::UniqueByArity::PLUGIN_TAG}#{message}") if config_proxy.debug_mode
|
30
|
+
end
|
31
|
+
|
32
|
+
# For per-class config with a block
|
33
|
+
def uniqueness_configure
|
34
|
+
@uniqueness_configuration ||= Configuration.new
|
35
|
+
yield(@uniqueness_configuration)
|
36
|
+
end
|
37
|
+
|
38
|
+
#### CONFIG ####
|
39
|
+
class << self
|
40
|
+
attr_accessor :uniqueness_configuration
|
41
|
+
end
|
42
|
+
self.uniqueness_configuration = Configuration.new # setup defaults
|
43
|
+
|
44
|
+
def uniqueness_config_reset(config = Configuration.new)
|
45
|
+
@uniqueness_configuration = config
|
46
|
+
end
|
47
|
+
|
48
|
+
def uniq_config
|
49
|
+
@uniqueness_configuration
|
50
|
+
end
|
51
|
+
end
|
52
|
+
end
|
53
|
+
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: resque-unique_by_arity
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 3.0.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Peter H. Boling
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2018-11-
|
11
|
+
date: 2018-11-15 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: colorize
|
@@ -216,7 +216,6 @@ extra_rdoc_files: []
|
|
216
216
|
files:
|
217
217
|
- ".gitignore"
|
218
218
|
- ".rspec"
|
219
|
-
- ".rspec_status"
|
220
219
|
- ".rubocop.yml"
|
221
220
|
- ".rubocop_todo.yml"
|
222
221
|
- ".ruby-version"
|
@@ -227,10 +226,13 @@ files:
|
|
227
226
|
- Rakefile
|
228
227
|
- bin/console
|
229
228
|
- bin/setup
|
229
|
+
- lib/resque-unique_by_arity.rb
|
230
230
|
- lib/resque/plugins/unique_by_arity.rb
|
231
231
|
- lib/resque/unique_by_arity.rb
|
232
232
|
- lib/resque/unique_by_arity/configuration.rb
|
233
|
+
- lib/resque/unique_by_arity/global_configuration.rb
|
233
234
|
- lib/resque/unique_by_arity/modulizer.rb
|
235
|
+
- lib/resque/unique_by_arity/unique_job.rb
|
234
236
|
- lib/resque/unique_by_arity/validation.rb
|
235
237
|
- lib/resque/unique_by_arity/version.rb
|
236
238
|
- resque-unique_by_arity.gemspec
|
data/.rspec_status
DELETED
@@ -1,55 +0,0 @@
|
|
1
|
-
example_id | status | run_time |
|
2
|
-
----------------------------------------------------------------- | ------ | --------------- |
|
3
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:1] | passed | 0.00313 seconds |
|
4
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:2:1] | passed | 0.00099 seconds |
|
5
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:3:1] | passed | 0.00038 seconds |
|
6
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:4:1] | passed | 0.00016 seconds |
|
7
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:5:1] | passed | 0.00017 seconds |
|
8
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:6:1] | passed | 0.00018 seconds |
|
9
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:7:1] | passed | 0.0002 seconds |
|
10
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:8:1] | passed | 0.00016 seconds |
|
11
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:9:1] | passed | 0.00014 seconds |
|
12
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:10:1] | passed | 0.00014 seconds |
|
13
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:11:1] | passed | 0.00017 seconds |
|
14
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:12:1] | passed | 0.00015 seconds |
|
15
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:13:1] | passed | 0.00017 seconds |
|
16
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:14:1] | passed | 0.00096 seconds |
|
17
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:14:2:1] | passed | 0.00014 seconds |
|
18
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:1:14:3:1] | passed | 0.00013 seconds |
|
19
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:2:1] | passed | 0.01014 seconds |
|
20
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:3:1] | passed | 0.00031 seconds |
|
21
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:3:2] | passed | 0.00024 seconds |
|
22
|
-
./spec/resque/unique_by_arity/configuration_spec.rb[1:1:4:1] | passed | 0.00017 seconds |
|
23
|
-
./spec/resque/unique_by_arity_spec.rb[1:1] | passed | 0.00006 seconds |
|
24
|
-
./spec/resque/unique_by_arity_spec.rb[1:2:1:1] | passed | 0.00094 seconds |
|
25
|
-
./spec/resque/unique_by_arity_spec.rb[1:3:1] | passed | 0.00062 seconds |
|
26
|
-
./spec/resque/unique_by_arity_spec.rb[1:4:1:1] | passed | 0.00022 seconds |
|
27
|
-
./spec/resque/unique_by_arity_spec.rb[1:5:1:1] | passed | 0.00086 seconds |
|
28
|
-
./spec/resque/unique_by_arity_spec.rb[1:5:2:1] | passed | 0.00026 seconds |
|
29
|
-
./spec/resque/unique_by_arity_spec.rb[1:5:3:1] | passed | 0.00027 seconds |
|
30
|
-
./spec/resque/unique_by_arity_spec.rb[1:6:1] | passed | 0.00047 seconds |
|
31
|
-
./spec/resque/unique_by_arity_spec.rb[1:7:1] | passed | 0.00021 seconds |
|
32
|
-
./spec/resque/unique_by_arity_spec.rb[1:7:2:1] | passed | 0.00057 seconds |
|
33
|
-
./spec/resque/unique_by_arity_spec.rb[1:7:3:1] | passed | 0.00062 seconds |
|
34
|
-
./spec/resque/unique_by_arity_spec.rb[1:7:4:1] | passed | 0.00021 seconds |
|
35
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:1:1:1:1] | passed | 0.00028 seconds |
|
36
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:1:2:1:1] | passed | 0.00025 seconds |
|
37
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:1:3:1:1] | passed | 0.00061 seconds |
|
38
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:2:1:1:1] | passed | 0.00022 seconds |
|
39
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:2:2:1:1] | passed | 0.00029 seconds |
|
40
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:2:3:1:1] | passed | 0.00057 seconds |
|
41
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:3:1:1:1] | passed | 0.00076 seconds |
|
42
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:3:2:1:1] | passed | 0.00075 seconds |
|
43
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:3:3:1:1] | passed | 0.00062 seconds |
|
44
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:4:1:1:1] | passed | 0.0004 seconds |
|
45
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:4:2:1:1] | passed | 0.00032 seconds |
|
46
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:4:3:1:1] | passed | 0.00042 seconds |
|
47
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:5:1:1:1] | passed | 0.00023 seconds |
|
48
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:5:2:1:1] | passed | 0.00029 seconds |
|
49
|
-
./spec/resque/unique_by_arity_spec.rb[1:8:5:3:1] | passed | 0.0002 seconds |
|
50
|
-
./spec/resque/unique_by_arity_spec.rb[1:9:1] | passed | 0.0025 seconds |
|
51
|
-
./spec/resque/unique_by_arity_spec.rb[1:10:1:1] | passed | 0.00012 seconds |
|
52
|
-
./spec/resque/unique_by_arity_spec.rb[1:11:1:1] | passed | 0.00029 seconds |
|
53
|
-
./spec/resque/unique_by_arity_spec.rb[1:11:2:1:1] | passed | 0.00025 seconds |
|
54
|
-
./spec/resque/unique_by_arity_spec.rb[1:11:2:2:1] | passed | 0.0006 seconds |
|
55
|
-
./spec/resque/unique_by_arity_spec.rb[1:11:2:3:1] | passed | 0.0005 seconds |
|