resque-workers-lock 1.0.3 → 1.0.4
Sign up to get free protection for your applications and to get access to all the features.
- data/README.md +10 -2
- data/lib/resque/plugins/workers/lock.rb +28 -10
- data/test/lock_test.rb +3 -2
- metadata +1 -1
data/README.md
CHANGED
@@ -12,7 +12,7 @@ If resque jobs have the same lock applied this means that those jobs cannot be p
|
|
12
12
|
By default the lock is the instance name + arguments (just like the classic resque-lock). Override this lock to lock on specific arguments.
|
13
13
|
|
14
14
|
## How does it differ from resque-lock?
|
15
|
-
Resque-lock will not let you queue jobs when you locked them. Resque-workers-lock locks on a workers-level and will requeue the locked jobs. Resque workers lock will not prevent you to queue jobs. If a worker takes on a job that is already being processed by another worker it will put the job back up in the queue!
|
15
|
+
Resque-lock will not let you queue jobs when you locked them. Resque-workers-lock locks on a workers-level and will requeue the locked jobs. Resque workers lock will not prevent you to queue jobs. If a worker takes on a job that is already being processed by another worker it will put the job back up in the queue! Also, this plugin will still offer you the ability to lock at enqueue-level just like resque-lock does (see example).
|
16
16
|
|
17
17
|
## Example
|
18
18
|
This example shows how you can use the workers-lock to prevent two jobs with the same domain to be processed simultaneously.
|
@@ -23,9 +23,14 @@ class Parser
|
|
23
23
|
extend Resque::Plugins::Workers::Lock
|
24
24
|
|
25
25
|
# Lock method has the same arguments as the self.perform
|
26
|
-
def self.
|
26
|
+
def self.workers_lock(domain, arg2, arg3)
|
27
27
|
return domain
|
28
28
|
end
|
29
|
+
|
30
|
+
# Turn off standard resque-lock functionality
|
31
|
+
def self.enqueue_lock(domain, arg2, arg3)
|
32
|
+
false
|
33
|
+
end
|
29
34
|
|
30
35
|
# Perform method with some arguments
|
31
36
|
def self.perform(domain, arg2, arg3)
|
@@ -35,6 +40,9 @@ end
|
|
35
40
|
```
|
36
41
|
In this example `domain` is used to specify certain types of jobs that are not allowed to run at the same time. For example: if you create three jobs with the domain argument google.com, google.com and yahoo.com, the two google.com jobs will never run at the same time.
|
37
42
|
|
43
|
+
## One queue
|
44
|
+
Best results with one big queue instead of multiple queues.
|
45
|
+
|
38
46
|
## Requeue loop
|
39
47
|
When a job is requeue'ed there is a small delay (1 second by default) before the worker places the job actually back in the queue. Let's say you have two jobs left, and one job is taking 15 seconds on the first worker and the other similar job is being blocked by the second worker. The second worker will continuously try to put the job back in the queue and it will try to process it again (racing for 15 seconds untill the other job has finished). This only happens when there are no other (not locked) jobs in the queue.
|
40
48
|
|
@@ -2,21 +2,36 @@ module Resque
|
|
2
2
|
module Plugins
|
3
3
|
module Workers
|
4
4
|
module Lock
|
5
|
-
|
6
|
-
|
7
|
-
|
5
|
+
|
6
|
+
# Override in your job to control the queue lock key
|
7
|
+
def enqueue_lock(*args)
|
8
|
+
"enqueuelock:#{name}-#{args.to_s}"
|
9
|
+
end
|
10
|
+
|
11
|
+
# Override in your job to control the workers lock key.
|
12
|
+
def workers_lock(*args)
|
13
|
+
"workerslock:#{name}-#{args.to_s}"
|
8
14
|
end
|
9
15
|
|
16
|
+
# Override in your job to change the perform requeue delay
|
10
17
|
def requeue_perform_delay
|
11
18
|
1.0
|
12
19
|
end
|
13
20
|
|
21
|
+
def before_enqueue_lock(*args)
|
22
|
+
if enqueue_lock(*args)
|
23
|
+
Resque.redis.setnx(enqueue_lock(*args), true)
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
14
27
|
def before_perform_lock(*args)
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
28
|
+
if workers_lock(*args)
|
29
|
+
nx = Resque.redis.setnx(workers_lock(*args), true)
|
30
|
+
if nx == false
|
31
|
+
sleep(requeue_perform_delay)
|
32
|
+
Resque.enqueue(self, *args)
|
33
|
+
raise Resque::Job::DontPerform
|
34
|
+
end
|
20
35
|
end
|
21
36
|
end
|
22
37
|
|
@@ -25,14 +40,17 @@ module Resque
|
|
25
40
|
yield
|
26
41
|
ensure
|
27
42
|
# Clear the lock. (even with errors)
|
28
|
-
Resque.redis.del(
|
43
|
+
Resque.redis.del(workers_lock(*args))
|
44
|
+
Resque.redis.del(enqueue_lock(*args))
|
29
45
|
end
|
30
46
|
end
|
31
47
|
|
32
48
|
def on_failure_lock(exception, *args)
|
33
49
|
# Clear the lock on DirtyExit
|
34
|
-
Resque.redis.del(
|
50
|
+
Resque.redis.del(workers_lock(*args))
|
51
|
+
Resque.redis.del(enqueue_lock(*args))
|
35
52
|
end
|
53
|
+
|
36
54
|
end
|
37
55
|
end
|
38
56
|
end
|
data/test/lock_test.rb
CHANGED
@@ -16,7 +16,8 @@ class LockTest < Test::Unit::TestCase
|
|
16
16
|
|
17
17
|
def setup
|
18
18
|
Resque.redis.del('queue:lock_test')
|
19
|
-
Resque.redis.del(Job.
|
19
|
+
Resque.redis.del(Job.workers_lock)
|
20
|
+
Resque.redis.del(Job.enqueue_lock)
|
20
21
|
end
|
21
22
|
|
22
23
|
def test_lint
|
@@ -35,7 +36,7 @@ class LockTest < Test::Unit::TestCase
|
|
35
36
|
def test_enqueue
|
36
37
|
3.times { Resque.enqueue(Job) }
|
37
38
|
|
38
|
-
assert_equal
|
39
|
+
assert_equal 1, Resque.redis.llen('queue:lock_test')
|
39
40
|
end
|
40
41
|
|
41
42
|
def test_lock
|