resque-unique_at_runtime 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.gitignore +17 -0
- data/.travis.yml +5 -0
- data/Gemfile +4 -0
- data/LICENSE +22 -0
- data/README.md +191 -0
- data/Rakefile +8 -0
- data/lib/resque-unique_at_runtime.rb +64 -0
- data/lib/resque-unique_at_runtime/version.rb +7 -0
- data/resque-unique_at_runtime.gemspec +42 -0
- data/spec/lib/unique_at_runtime_spec.rb +180 -0
- data/spec/spec_helper.rb +14 -0
- metadata +145 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 5dac074b3279cfd0d0e3eaf495ead3828b3eaf73
|
4
|
+
data.tar.gz: 2e87da59e0cc4c4af1843659e19638d1365d90ab
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 1b6c50a917f58b0049d79264ae28f1258514d90b6e1ea23d8435f521491b1c2ce48803aad5423747b48ed561c936ad601b5b60e1a11c34bc3d02e3512b03c5e5
|
7
|
+
data.tar.gz: 863df5e7e568db23ab2509c1b0b9e3ddf36b2d8b555239cd462508923b7277e8a43e90563a9fe97718e2f7d5a345978c9aa5203fd9d72fb06a71d7f58e4e809b
|
data/.gitignore
ADDED
data/.travis.yml
ADDED
data/Gemfile
ADDED
data/LICENSE
ADDED
@@ -0,0 +1,22 @@
|
|
1
|
+
Copyright (c) 2012 Jonathan R. Wallace
|
2
|
+
|
3
|
+
MIT License
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
6
|
+
a copy of this software and associated documentation files (the
|
7
|
+
"Software"), to deal in the Software without restriction, including
|
8
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
9
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
10
|
+
permit persons to whom the Software is furnished to do so, subject to
|
11
|
+
the following conditions:
|
12
|
+
|
13
|
+
The above copyright notice and this permission notice shall be
|
14
|
+
included in all copies or substantial portions of the Software.
|
15
|
+
|
16
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
17
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
18
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
19
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
20
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
21
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
22
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,191 @@
|
|
1
|
+
# Resque::Plugins::UniqueAtRuntime
|
2
|
+
|
3
|
+
[](https://travis-ci.org/pboling/resque-lonely\_job)
|
4
|
+
|
5
|
+
A [semanticaly versioned](http://semver.org/)
|
6
|
+
[Resque](https://github.com/resque/resque) plugin which ensures for a given
|
7
|
+
queue, that only one worker is working on a job at any given time.
|
8
|
+
|
9
|
+
Resque::Plugins::UniqueAtRuntime differs from [resque-lonely_job](https://github.com/wallace/resque-lonely_job) in that it is compatible with, and can be used at the same time as, [resque-solo](https://github.com/neighborland/resque_solo).
|
10
|
+
|
11
|
+
Resque::Plugins::UniqueAtRuntime differs from [resque_solo](https://github.com/neighborland/resque_solo) in that `resque-solo` offers **queue-time** uniqueness, while `resque-unique_at_runtime` offers **runtime** uniqueness. The same difference applies to other queue-time uniqueness gems: [resque-queue-lock](https://github.com/mashion/resque-queue-lock), [resque-lock](https://github.com/defunkt/resque-lock).
|
12
|
+
|
13
|
+
Runtime uniqueness without queue-time uniqueness means the same job may be queued multiple times but you're guaranteed that first job queued will run to completion before subsequent jobs are run.
|
14
|
+
|
15
|
+
However, you can use both runtime and queue-time uniqueness together in the same project.
|
16
|
+
|
17
|
+
To use `resque-solo` and `resque-unique_at_runtime` together, with fine control of per job configuration of uniqueness at runtime and queue-time, it is recommended to use [resque-unique_by_arity](https://github.com/pboling/resque-unique_by_arity).
|
18
|
+
|
19
|
+
NOTE: There is a *strong* possibility that subsequent jobs are re-ordered due to
|
20
|
+
the implementation of
|
21
|
+
[reenqueue](https://github.com/pboling/resque-unique_at_runtime/blob/master/lib/resque-unique_at_runtime.rb#L35).
|
22
|
+
(See Example #2 for an alternative approach that attempts to preserve job
|
23
|
+
ordering but introduces the possibility of starvation.)
|
24
|
+
|
25
|
+
Therefore it is recommended that the payload for jobs be stored in a separate
|
26
|
+
redis list distinct from the Resque queue (see Example #3).
|
27
|
+
|
28
|
+
## Requirements
|
29
|
+
|
30
|
+
Requires a version of MRI Ruby >= 1.9.3.
|
31
|
+
|
32
|
+
## Installation
|
33
|
+
|
34
|
+
Add this line to your application's Gemfile:
|
35
|
+
|
36
|
+
gem 'resque-unique_at_runtime', '~> 1.0.0'
|
37
|
+
|
38
|
+
And then execute:
|
39
|
+
|
40
|
+
$ bundle
|
41
|
+
|
42
|
+
Or install it yourself as:
|
43
|
+
|
44
|
+
$ gem install resque-unique_at_runtime
|
45
|
+
|
46
|
+
## Usage
|
47
|
+
|
48
|
+
#### Example #1 -- One job running per queue
|
49
|
+
|
50
|
+
require 'resque-unique_at_runtime'
|
51
|
+
|
52
|
+
class StrictlySerialJob
|
53
|
+
extend Resque::Plugins::UniqueAtRuntime
|
54
|
+
|
55
|
+
@queue = :serial_work
|
56
|
+
|
57
|
+
def self.perform
|
58
|
+
# only one at a time in this block, no parallelism allowed for this
|
59
|
+
# particular queue
|
60
|
+
end
|
61
|
+
end
|
62
|
+
|
63
|
+
#### Example #2 -- One job running per user-defined attribute
|
64
|
+
|
65
|
+
Let's say you want the serial constraint to apply at a more granular
|
66
|
+
level. Instead of applying at the queue level, you can overwrite the .redis\_key
|
67
|
+
method.
|
68
|
+
|
69
|
+
require 'resque-unique_at_runtime'
|
70
|
+
|
71
|
+
class StrictlySerialJob
|
72
|
+
extend Resque::Plugins::UniqueAtRuntime
|
73
|
+
|
74
|
+
@queue = :serial_work
|
75
|
+
|
76
|
+
# Returns a string that will be used as the redis key
|
77
|
+
# NOTE: it is recommended to prefix your string with the 'unique_at_runtime:' to
|
78
|
+
# namespace your key!
|
79
|
+
def self.unique_at_runtime_redis_key(account_id, *args)
|
80
|
+
"unique_at_runtime:strictly_serial_job:#{account_id}"
|
81
|
+
end
|
82
|
+
|
83
|
+
# Overwrite reenqueue to lpush instead of default rpush. This attempts to
|
84
|
+
# preserve job ordering but job order is *NOT* guaranteed and also not
|
85
|
+
# likely. See the comment on SHA: e9912fb2 for why.
|
86
|
+
def self.reenqueue(*args)
|
87
|
+
Resque.redis.lpush("queue:#{Resque.queue_from_class(self)}", Resque.encode(class: self, args: args))
|
88
|
+
end
|
89
|
+
|
90
|
+
def self.perform(account_id, *args)
|
91
|
+
# only one at a time in this block, no parallelism allowed for this
|
92
|
+
# particular unique_at_runtime_redis_key
|
93
|
+
end
|
94
|
+
end
|
95
|
+
|
96
|
+
*NOTE*: Without careful consideration of your problem domain, worker starvation
|
97
|
+
and/or unfairness is possible for jobs in this example. Imagine a scenario
|
98
|
+
where you have three jobs in the queue with two resque workers:
|
99
|
+
|
100
|
+
+---------------------------------------------------+
|
101
|
+
| :serial_work |
|
102
|
+
|---------------------------------------------------|
|
103
|
+
| | | | |
|
104
|
+
| unique_at_runtime_redis_key: | unique_at_runtime_redis_key: | unique_at_runtime_redis_key: | ... |
|
105
|
+
| A | A | B | |
|
106
|
+
| | | | |
|
107
|
+
| job 1 | job 2 | job 3 | |
|
108
|
+
+---------------------------------------------------+
|
109
|
+
^
|
110
|
+
|
|
111
|
+
Possible starvation +-----------+
|
112
|
+
for this job and
|
113
|
+
subsequent ones
|
114
|
+
|
115
|
+
|
116
|
+
When the first worker grabs job 1, it'll acquire the mutex for processing
|
117
|
+
redis\_key A. The second worker tries to grab the next job off the queue but
|
118
|
+
is unable to acquire the mutex for redis\_key A so it places job 2 back at the
|
119
|
+
head of the :serial\_work queue. Until worker 1 completes job 1 and releases
|
120
|
+
the mutex for redis\_key A, no work will be done in this queue.
|
121
|
+
|
122
|
+
This issue may be avoided by employing dynamic queues,
|
123
|
+
http://blog.kabisa.nl/2010/03/16/dynamic-queue-assignment-for-resque-jobs/,
|
124
|
+
where the queue is a one to one mapping to the redis\_key.
|
125
|
+
|
126
|
+
#### Example #3 -- One job running per user-defined attribute with job ordering preserved
|
127
|
+
|
128
|
+
The secret to preserving job order semantics is to remove critical data from the
|
129
|
+
resque job and store data in a separate redis list. Part of a running job's
|
130
|
+
responsibility will be to grab data off of the separate redis list needed for it
|
131
|
+
to complete its job.
|
132
|
+
|
133
|
+
+---------------------------------------------------+
|
134
|
+
| :serial_work for jobs associated with key A |
|
135
|
+
|---------------------------------------------------|
|
136
|
+
| data x | data y | data z | ... |
|
137
|
+
+---------------------------------------------------+
|
138
|
+
|
139
|
+
+---------------------------------------------------+
|
140
|
+
| :serial_work for jobs associated with key B |
|
141
|
+
|---------------------------------------------------|
|
142
|
+
| data m | data n | data o | ... |
|
143
|
+
+---------------------------------------------------+
|
144
|
+
|
145
|
+
+---------------------------------------------------+
|
146
|
+
| :serial_work |
|
147
|
+
|---------------------------------------------------|
|
148
|
+
| | | | |
|
149
|
+
| unique_at_runtime_redis_key: | unique_at_runtime_redis_key: | unique_at_runtime_redis_key: | ... |
|
150
|
+
| A | A | B | |
|
151
|
+
| | | | |
|
152
|
+
| job 1 | job 2 | job 3 | |
|
153
|
+
+---------------------------------------------------+
|
154
|
+
|
155
|
+
It now doesn't matter whether job 1 and job 2 are re-ordered as whichever goes
|
156
|
+
first will perform an atomic pop on the redis list that contains the data needed
|
157
|
+
for its job (data x, data y, data z).
|
158
|
+
|
159
|
+
#### Example #4 -- Requeue interval
|
160
|
+
|
161
|
+
The behavior when multiple jobs exist in a queue protected by resque-unique_at_runtime
|
162
|
+
is for one job to be worked, while the other is continuously dequeued and
|
163
|
+
requeued until the first job is finished. This can result in that worker
|
164
|
+
process pegging a CPU/core on a worker server. To guard against this, the
|
165
|
+
default behavior is to sleep for 1 second before the requeue, which will allow
|
166
|
+
the cpu to perform other work.
|
167
|
+
|
168
|
+
This can be customized using a ```@requeue_interval``` class instance variable
|
169
|
+
in your job like so:
|
170
|
+
|
171
|
+
|
172
|
+
require 'resque-unique_at_runtime'
|
173
|
+
|
174
|
+
class StrictlySerialJob
|
175
|
+
extend Resque::Plugins::UniqueAtRuntime
|
176
|
+
|
177
|
+
@queue = :serial_work
|
178
|
+
@requeue_interval = 5 # sleep for 5 seconds before requeueing
|
179
|
+
|
180
|
+
def self.perform
|
181
|
+
# some implementation
|
182
|
+
end
|
183
|
+
end
|
184
|
+
|
185
|
+
## Contributing
|
186
|
+
|
187
|
+
1. Fork it
|
188
|
+
2. Create your feature branch (`git checkout -b my-new-feature`)
|
189
|
+
3. Commit your changes (`git commit -am 'Added some feature'`)
|
190
|
+
4. Push to the branch (`git push origin my-new-feature`)
|
191
|
+
5. Create new Pull Request
|
data/Rakefile
ADDED
@@ -0,0 +1,64 @@
|
|
1
|
+
require 'resque-unique_at_runtime/version'
|
2
|
+
|
3
|
+
module Resque
|
4
|
+
module Plugins
|
5
|
+
module UniqueAtRuntime
|
6
|
+
LOCK_TIMEOUT = 60 * 60 * 24 * 5 # 5 days
|
7
|
+
|
8
|
+
def lock_timeout
|
9
|
+
Time.now.to_i + LOCK_TIMEOUT + 1
|
10
|
+
end
|
11
|
+
|
12
|
+
def requeue_interval
|
13
|
+
self.instance_variable_get(:@requeue_interval) || 1
|
14
|
+
end
|
15
|
+
|
16
|
+
# Overwrite this method to uniquely identify which mutex should be used
|
17
|
+
# for a resque worker.
|
18
|
+
def unique_at_runtime_redis_key(*_)
|
19
|
+
"unique_at_runtime:#{@queue}"
|
20
|
+
end
|
21
|
+
|
22
|
+
def can_lock_queue?(*args)
|
23
|
+
now = Time.now.to_i
|
24
|
+
key = unique_at_runtime_redis_key(*args)
|
25
|
+
timeout = lock_timeout
|
26
|
+
|
27
|
+
# Per http://redis.io/commands/setnx
|
28
|
+
return true if Resque.redis.setnx(key, timeout)
|
29
|
+
return false if Resque.redis.get(key).to_i > now
|
30
|
+
return true if Resque.redis.getset(key, timeout).to_i <= now
|
31
|
+
return false
|
32
|
+
end
|
33
|
+
|
34
|
+
def unlock_queue(*args)
|
35
|
+
Resque.redis.del(unique_at_runtime_redis_key(*args))
|
36
|
+
end
|
37
|
+
|
38
|
+
def reenqueue(*args)
|
39
|
+
Resque.enqueue(self, *args)
|
40
|
+
end
|
41
|
+
|
42
|
+
def before_perform(*args)
|
43
|
+
unless can_lock_queue?(*args)
|
44
|
+
# Sleep so the CPU's rest
|
45
|
+
sleep(requeue_interval)
|
46
|
+
|
47
|
+
# can't get the lock, so re-enqueue the task
|
48
|
+
reenqueue(*args)
|
49
|
+
|
50
|
+
# and don't perform
|
51
|
+
raise Resque::Job::DontPerform
|
52
|
+
end
|
53
|
+
end
|
54
|
+
|
55
|
+
def around_perform(*args)
|
56
|
+
begin
|
57
|
+
yield
|
58
|
+
ensure
|
59
|
+
unlock_queue(*args)
|
60
|
+
end
|
61
|
+
end
|
62
|
+
end
|
63
|
+
end
|
64
|
+
end
|
@@ -0,0 +1,42 @@
|
|
1
|
+
# -*- encoding: utf-8 -*-
|
2
|
+
require File.expand_path('../lib/resque-unique_at_runtime/version', __FILE__)
|
3
|
+
|
4
|
+
Gem::Specification.new do |gem|
|
5
|
+
gem.authors = ["Peter H. Boling","Jonathan R. Wallace"]
|
6
|
+
gem.email = ["peter.boling@gmail.com","jonathan.wallace@gmail.com"]
|
7
|
+
gem.summary = %q{A resque plugin that ensures job uniqueness at runtime.}
|
8
|
+
gem.homepage = "http://github.com/pboling/resque-unique_at_runtime"
|
9
|
+
|
10
|
+
gem.files = `git ls-files`.split($\)
|
11
|
+
gem.executables = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
|
12
|
+
gem.test_files = gem.files.grep(%r{^(test|spec|features)/})
|
13
|
+
gem.name = "resque-unique_at_runtime"
|
14
|
+
gem.require_paths = ["lib"]
|
15
|
+
gem.version = Resque::Plugins::UniqueAtRuntime::VERSION
|
16
|
+
gem.license = "MIT"
|
17
|
+
|
18
|
+
gem.add_dependency 'resque', '>= 1.2'
|
19
|
+
gem.add_development_dependency 'mock_redis'
|
20
|
+
gem.add_development_dependency 'rake'
|
21
|
+
gem.add_development_dependency 'rspec', '>= 3.0'
|
22
|
+
gem.add_development_dependency 'timecop'
|
23
|
+
|
24
|
+
gem.description = <<desc
|
25
|
+
Ensures that for a given queue, only one worker is working on a job at any given time.
|
26
|
+
|
27
|
+
Example:
|
28
|
+
|
29
|
+
require 'resque/plugins/unique_at_runtime'
|
30
|
+
|
31
|
+
class StrictlySerialJob
|
32
|
+
extend Resque::Plugins::UniqueAtRuntime
|
33
|
+
|
34
|
+
@queue = :serial_work
|
35
|
+
|
36
|
+
def self.perform
|
37
|
+
# only one at a time in this block, no parallelism allowed for this
|
38
|
+
# particular queue
|
39
|
+
end
|
40
|
+
end
|
41
|
+
desc
|
42
|
+
end
|
@@ -0,0 +1,180 @@
|
|
1
|
+
require 'spec_helper'
|
2
|
+
|
3
|
+
class SerialJob
|
4
|
+
extend Resque::Plugins::UniqueAtRuntime
|
5
|
+
@queue = :serial_work
|
6
|
+
|
7
|
+
def self.perform(*args); end
|
8
|
+
end
|
9
|
+
|
10
|
+
class SerialJobWithCustomRedisKey
|
11
|
+
extend Resque::Plugins::UniqueAtRuntime
|
12
|
+
@queue = :serial_work
|
13
|
+
|
14
|
+
def self.unique_at_runtime_redis_key(account_id, *args)
|
15
|
+
"unique_at_runtime:#{@queue}:#{account_id}"
|
16
|
+
end
|
17
|
+
|
18
|
+
def self.perform(account_id, *args); end
|
19
|
+
end
|
20
|
+
|
21
|
+
describe Resque::Plugins::UniqueAtRuntime do
|
22
|
+
before do
|
23
|
+
Resque.redis.flushall
|
24
|
+
end
|
25
|
+
|
26
|
+
describe ".requeue_interval" do
|
27
|
+
it "should default to 5" do
|
28
|
+
expect(SerialJob.requeue_interval).to eql(1)
|
29
|
+
end
|
30
|
+
|
31
|
+
it "should be overridable with a class instance var" do
|
32
|
+
SerialJob.instance_variable_set(:@requeue_interval, 5)
|
33
|
+
expect(SerialJob.requeue_interval).to eql(5)
|
34
|
+
end
|
35
|
+
end
|
36
|
+
|
37
|
+
describe ".can_lock_queue?" do
|
38
|
+
it 'can lock a queue' do
|
39
|
+
expect(SerialJob.can_lock_queue?(:serial_work)).to eql(true)
|
40
|
+
end
|
41
|
+
|
42
|
+
it 'cannot lock an already locked queue' do
|
43
|
+
expect(SerialJob.can_lock_queue?(:serial_work)).to eql(true)
|
44
|
+
expect(SerialJob.can_lock_queue?(:serial_work)).to eql(false)
|
45
|
+
end
|
46
|
+
|
47
|
+
it 'cannot lock a queue with active lock' do
|
48
|
+
expect(SerialJob.can_lock_queue?(:serial_work)).to eql(true)
|
49
|
+
Timecop.travel(Date.today + 1) do
|
50
|
+
expect(SerialJob.can_lock_queue?(:serial_work)).to eql(false)
|
51
|
+
end
|
52
|
+
end
|
53
|
+
|
54
|
+
it 'can relock a queue with expired lock' do
|
55
|
+
expect(SerialJob.can_lock_queue?(:serial_work)).to eql(true)
|
56
|
+
|
57
|
+
Timecop.travel(Date.today + 10) do
|
58
|
+
expect(SerialJob.can_lock_queue?(:serial_work)).to eql(true)
|
59
|
+
end
|
60
|
+
end
|
61
|
+
|
62
|
+
it 'solves race condition with getset' do
|
63
|
+
expect(SerialJob.can_lock_queue?(:serial_work)).to eql(true)
|
64
|
+
|
65
|
+
Timecop.travel(Date.today + 10) do
|
66
|
+
threads = (1..10).to_a.map {
|
67
|
+
Thread.new {
|
68
|
+
Thread.current[:locked] = SerialJob.can_lock_queue?(:serial_work)
|
69
|
+
}
|
70
|
+
}
|
71
|
+
|
72
|
+
# Only one worker should acquire lock
|
73
|
+
locks = threads.map {|t| t.join; t[:locked] }
|
74
|
+
expect(locks.count(true)).to eql(1)
|
75
|
+
end
|
76
|
+
end
|
77
|
+
end
|
78
|
+
|
79
|
+
describe ".perform" do
|
80
|
+
before do
|
81
|
+
SerialJob.instance_variable_set(:@requeue_interval, 0)
|
82
|
+
end
|
83
|
+
|
84
|
+
describe "using the default redis key" do
|
85
|
+
it 'should lock and unlock the queue' do
|
86
|
+
job = Resque::Job.new(:serial_work, { 'class' => 'SerialJob', 'args' => %w[account_one job_one] })
|
87
|
+
|
88
|
+
# job is the first SerialJob to run so it can lock the queue and perform
|
89
|
+
expect(SerialJob).to receive(:can_lock_queue?).and_return(true)
|
90
|
+
|
91
|
+
# but it should also clean up after itself
|
92
|
+
expect(SerialJob).to receive(:unlock_queue)
|
93
|
+
|
94
|
+
job.perform
|
95
|
+
end
|
96
|
+
|
97
|
+
it 'should clean up lock even with catastrophic job failure' do
|
98
|
+
job = Resque::Job.new(:serial_work, { 'class' => 'SerialJob', 'args' => %w[account_one job_one] })
|
99
|
+
|
100
|
+
# job is the first SerialJob to run so it can lock the queue and perform
|
101
|
+
expect(SerialJob).to receive(:can_lock_queue?).and_return(true)
|
102
|
+
|
103
|
+
# but we have a catastrophic job failure
|
104
|
+
expect(SerialJob).to receive(:perform).and_raise(Exception)
|
105
|
+
|
106
|
+
# and still it should clean up after itself
|
107
|
+
expect(SerialJob).to receive(:unlock_queue)
|
108
|
+
|
109
|
+
# unfortunately, the job will be lost but resque doesn't guarantee jobs
|
110
|
+
# aren't lost
|
111
|
+
expect { job.perform }.to raise_error(Exception)
|
112
|
+
end
|
113
|
+
|
114
|
+
it 'should place self at the end of the queue if unable to acquire the lock' do
|
115
|
+
job1_payload = %w[account_one job_one]
|
116
|
+
job2_payload = %w[account_one job_two]
|
117
|
+
Resque::Job.create(:serial_work, 'SerialJob', job1_payload)
|
118
|
+
Resque::Job.create(:serial_work, 'SerialJob', job2_payload)
|
119
|
+
|
120
|
+
expect(SerialJob).to receive(:can_lock_queue?).and_return(false)
|
121
|
+
|
122
|
+
# perform returns false when DontPerform exception is raised in
|
123
|
+
# before_perform callback
|
124
|
+
job1 = Resque.reserve(:serial_work)
|
125
|
+
expect(job1.perform).to eql(false)
|
126
|
+
|
127
|
+
first_queue_element = Resque.reserve(:serial_work)
|
128
|
+
expect(first_queue_element.payload["args"]).to eql([job2_payload])
|
129
|
+
end
|
130
|
+
end
|
131
|
+
|
132
|
+
describe "with a custom unique_at_runtime_redis_key" do
|
133
|
+
it 'should lock and unlock the queue' do
|
134
|
+
job = Resque::Job.new(:serial_work, { 'class' => 'SerialJobWithCustomRedisKey', 'args' => %w[account_one job_one] })
|
135
|
+
|
136
|
+
# job is the first SerialJobWithCustomRedisKey to run so it can lock the queue and perform
|
137
|
+
expect(SerialJobWithCustomRedisKey).to receive(:can_lock_queue?).and_return(true)
|
138
|
+
|
139
|
+
# but it should also clean up after itself
|
140
|
+
expect(SerialJobWithCustomRedisKey).to receive(:unlock_queue)
|
141
|
+
|
142
|
+
job.perform
|
143
|
+
end
|
144
|
+
|
145
|
+
it 'should clean up lock even with catastrophic job failure' do
|
146
|
+
job = Resque::Job.new(:serial_work, { 'class' => 'SerialJobWithCustomRedisKey', 'args' => %w[account_one job_one] })
|
147
|
+
|
148
|
+
# job is the first SerialJobWithCustomRedisKey to run so it can lock the queue and perform
|
149
|
+
expect(SerialJobWithCustomRedisKey).to receive(:can_lock_queue?).and_return(true)
|
150
|
+
|
151
|
+
# but we have a catastrophic job failure
|
152
|
+
expect(SerialJobWithCustomRedisKey).to receive(:perform).and_raise(Exception)
|
153
|
+
|
154
|
+
# and still it should clean up after itself
|
155
|
+
expect(SerialJobWithCustomRedisKey).to receive(:unlock_queue)
|
156
|
+
|
157
|
+
# unfortunately, the job will be lost but resque doesn't guarantee jobs
|
158
|
+
# aren't lost
|
159
|
+
expect { job.perform }.to raise_error(Exception)
|
160
|
+
end
|
161
|
+
|
162
|
+
it 'should place self at the end of the queue if unable to acquire the lock' do
|
163
|
+
job1_payload = %w[account_one job_one]
|
164
|
+
job2_payload = %w[account_one job_two]
|
165
|
+
Resque::Job.create(:serial_work, 'SerialJobWithCustomRedisKey', job1_payload)
|
166
|
+
Resque::Job.create(:serial_work, 'SerialJobWithCustomRedisKey', job2_payload)
|
167
|
+
|
168
|
+
expect(SerialJobWithCustomRedisKey).to receive(:can_lock_queue?).and_return(false)
|
169
|
+
|
170
|
+
# perform returns false when DontPerform exception is raised in
|
171
|
+
# before_perform callback
|
172
|
+
job1 = Resque.reserve(:serial_work)
|
173
|
+
expect(job1.perform).to eql(false)
|
174
|
+
|
175
|
+
first_queue_element = Resque.reserve(:serial_work)
|
176
|
+
expect(first_queue_element.payload["args"]).to eql([job2_payload])
|
177
|
+
end
|
178
|
+
end
|
179
|
+
end
|
180
|
+
end
|
data/spec/spec_helper.rb
ADDED
metadata
ADDED
@@ -0,0 +1,145 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: resque-unique_at_runtime
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 2.0.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Peter H. Boling
|
8
|
+
- Jonathan R. Wallace
|
9
|
+
autorequire:
|
10
|
+
bindir: bin
|
11
|
+
cert_chain: []
|
12
|
+
date: 2017-10-01 00:00:00.000000000 Z
|
13
|
+
dependencies:
|
14
|
+
- !ruby/object:Gem::Dependency
|
15
|
+
name: resque
|
16
|
+
requirement: !ruby/object:Gem::Requirement
|
17
|
+
requirements:
|
18
|
+
- - ">="
|
19
|
+
- !ruby/object:Gem::Version
|
20
|
+
version: '1.2'
|
21
|
+
type: :runtime
|
22
|
+
prerelease: false
|
23
|
+
version_requirements: !ruby/object:Gem::Requirement
|
24
|
+
requirements:
|
25
|
+
- - ">="
|
26
|
+
- !ruby/object:Gem::Version
|
27
|
+
version: '1.2'
|
28
|
+
- !ruby/object:Gem::Dependency
|
29
|
+
name: mock_redis
|
30
|
+
requirement: !ruby/object:Gem::Requirement
|
31
|
+
requirements:
|
32
|
+
- - ">="
|
33
|
+
- !ruby/object:Gem::Version
|
34
|
+
version: '0'
|
35
|
+
type: :development
|
36
|
+
prerelease: false
|
37
|
+
version_requirements: !ruby/object:Gem::Requirement
|
38
|
+
requirements:
|
39
|
+
- - ">="
|
40
|
+
- !ruby/object:Gem::Version
|
41
|
+
version: '0'
|
42
|
+
- !ruby/object:Gem::Dependency
|
43
|
+
name: rake
|
44
|
+
requirement: !ruby/object:Gem::Requirement
|
45
|
+
requirements:
|
46
|
+
- - ">="
|
47
|
+
- !ruby/object:Gem::Version
|
48
|
+
version: '0'
|
49
|
+
type: :development
|
50
|
+
prerelease: false
|
51
|
+
version_requirements: !ruby/object:Gem::Requirement
|
52
|
+
requirements:
|
53
|
+
- - ">="
|
54
|
+
- !ruby/object:Gem::Version
|
55
|
+
version: '0'
|
56
|
+
- !ruby/object:Gem::Dependency
|
57
|
+
name: rspec
|
58
|
+
requirement: !ruby/object:Gem::Requirement
|
59
|
+
requirements:
|
60
|
+
- - ">="
|
61
|
+
- !ruby/object:Gem::Version
|
62
|
+
version: '3.0'
|
63
|
+
type: :development
|
64
|
+
prerelease: false
|
65
|
+
version_requirements: !ruby/object:Gem::Requirement
|
66
|
+
requirements:
|
67
|
+
- - ">="
|
68
|
+
- !ruby/object:Gem::Version
|
69
|
+
version: '3.0'
|
70
|
+
- !ruby/object:Gem::Dependency
|
71
|
+
name: timecop
|
72
|
+
requirement: !ruby/object:Gem::Requirement
|
73
|
+
requirements:
|
74
|
+
- - ">="
|
75
|
+
- !ruby/object:Gem::Version
|
76
|
+
version: '0'
|
77
|
+
type: :development
|
78
|
+
prerelease: false
|
79
|
+
version_requirements: !ruby/object:Gem::Requirement
|
80
|
+
requirements:
|
81
|
+
- - ">="
|
82
|
+
- !ruby/object:Gem::Version
|
83
|
+
version: '0'
|
84
|
+
description: |
|
85
|
+
Ensures that for a given queue, only one worker is working on a job at any given time.
|
86
|
+
|
87
|
+
Example:
|
88
|
+
|
89
|
+
require 'resque/plugins/unique_at_runtime'
|
90
|
+
|
91
|
+
class StrictlySerialJob
|
92
|
+
extend Resque::Plugins::UniqueAtRuntime
|
93
|
+
|
94
|
+
@queue = :serial_work
|
95
|
+
|
96
|
+
def self.perform
|
97
|
+
# only one at a time in this block, no parallelism allowed for this
|
98
|
+
# particular queue
|
99
|
+
end
|
100
|
+
end
|
101
|
+
email:
|
102
|
+
- peter.boling@gmail.com
|
103
|
+
- jonathan.wallace@gmail.com
|
104
|
+
executables: []
|
105
|
+
extensions: []
|
106
|
+
extra_rdoc_files: []
|
107
|
+
files:
|
108
|
+
- ".gitignore"
|
109
|
+
- ".travis.yml"
|
110
|
+
- Gemfile
|
111
|
+
- LICENSE
|
112
|
+
- README.md
|
113
|
+
- Rakefile
|
114
|
+
- lib/resque-unique_at_runtime.rb
|
115
|
+
- lib/resque-unique_at_runtime/version.rb
|
116
|
+
- resque-unique_at_runtime.gemspec
|
117
|
+
- spec/lib/unique_at_runtime_spec.rb
|
118
|
+
- spec/spec_helper.rb
|
119
|
+
homepage: http://github.com/pboling/resque-unique_at_runtime
|
120
|
+
licenses:
|
121
|
+
- MIT
|
122
|
+
metadata: {}
|
123
|
+
post_install_message:
|
124
|
+
rdoc_options: []
|
125
|
+
require_paths:
|
126
|
+
- lib
|
127
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
128
|
+
requirements:
|
129
|
+
- - ">="
|
130
|
+
- !ruby/object:Gem::Version
|
131
|
+
version: '0'
|
132
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
133
|
+
requirements:
|
134
|
+
- - ">="
|
135
|
+
- !ruby/object:Gem::Version
|
136
|
+
version: '0'
|
137
|
+
requirements: []
|
138
|
+
rubyforge_project:
|
139
|
+
rubygems_version: 2.6.12
|
140
|
+
signing_key:
|
141
|
+
specification_version: 4
|
142
|
+
summary: A resque plugin that ensures job uniqueness at runtime.
|
143
|
+
test_files:
|
144
|
+
- spec/lib/unique_at_runtime_spec.rb
|
145
|
+
- spec/spec_helper.rb
|