resque-lock-timeout 0.2.0
Sign up to get free protection for your applications and to get access to all the features.
- data/LICENSE +22 -0
- data/README.md +138 -0
- data/Rakefile +21 -0
- data/lib/resque-lock-timeout.rb +1 -0
- data/lib/resque/plugins/lock_timeout.rb +151 -0
- data/test/lock_test.rb +108 -0
- data/test/redis-test.conf +132 -0
- data/test/test_helper.rb +41 -0
- data/test/test_jobs.rb +62 -0
- metadata +109 -0
data/LICENSE
ADDED
@@ -0,0 +1,22 @@
|
|
1
|
+
Copyright (c) 2010 Chris Wanstrath
|
2
|
+
Copyright (c) 2010 Ryan Carver
|
3
|
+
Copyright (c) 2010 Luke Antins
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
6
|
+
a copy of this software and associated documentation files (the
|
7
|
+
Software), to deal in the Software without restriction, including
|
8
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
9
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
10
|
+
permit persons to whom the Software is furnished to do so, subject to
|
11
|
+
the following conditions:
|
12
|
+
|
13
|
+
The above copyright notice and this permission notice shall be
|
14
|
+
included in all copies or substantial portions of the Software.
|
15
|
+
|
16
|
+
THE SOFTWARE IS PROVIDED AS IS, WITHOUT WARRANTY OF ANY KIND,
|
17
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
18
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
19
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
20
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
21
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
22
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,138 @@
|
|
1
|
+
Resque Lock Timeout
|
2
|
+
===================
|
3
|
+
|
4
|
+
A [Resque][rq] plugin. Requires Resque 1.8.0.
|
5
|
+
|
6
|
+
resque-lock-timeout adds locking, with optional timeout/deadlock handling to
|
7
|
+
resque jobs.
|
8
|
+
|
9
|
+
Using a `lock_timeout` allows you to re-aquire the lock should your worker
|
10
|
+
fail, crash, or is otherwise unable to relase the lock. **i.e.** Your server
|
11
|
+
unexpectedly looses power. Very handy for jobs that are recurring or may be
|
12
|
+
retried.
|
13
|
+
|
14
|
+
Usage / Examples
|
15
|
+
----------------
|
16
|
+
|
17
|
+
### Single Job Instance
|
18
|
+
|
19
|
+
require 'resque-lock-timeout'
|
20
|
+
|
21
|
+
class UpdateNetworkGraph
|
22
|
+
extend Resque::Plugins::LockTimeout
|
23
|
+
@queue = :network_graph
|
24
|
+
|
25
|
+
def self.perform(repo_id)
|
26
|
+
heavy_lifting
|
27
|
+
end
|
28
|
+
end
|
29
|
+
|
30
|
+
Locking is achieved by storing a identifyer/lock key in Redis.
|
31
|
+
|
32
|
+
Default behaviour...
|
33
|
+
|
34
|
+
* Only one instance of a job may execute at once.
|
35
|
+
* The lock is held until the job completes or fails.
|
36
|
+
* If another job is executing with the same arguments the job will abort.
|
37
|
+
|
38
|
+
Please see below for more information about the identifer/lock key.
|
39
|
+
|
40
|
+
### With Lock Expiry/Timeout
|
41
|
+
|
42
|
+
The locking algorithm used can be found in the [Redis SETNX][redis-setnx]
|
43
|
+
documentation.
|
44
|
+
|
45
|
+
Simply set the lock timeout in seconds, e.g.
|
46
|
+
|
47
|
+
class UpdateNetworkGraph
|
48
|
+
extend Resque::Plugins::LockTimeout
|
49
|
+
@queue = :network_graph
|
50
|
+
|
51
|
+
# Lock may be held for upto an hour.
|
52
|
+
@lock_timeout = 3600
|
53
|
+
|
54
|
+
def self.perform(repo_id)
|
55
|
+
heavy_lifting
|
56
|
+
end
|
57
|
+
end
|
58
|
+
|
59
|
+
Customise & Extend
|
60
|
+
==================
|
61
|
+
|
62
|
+
### Job Identifier/Lock Key
|
63
|
+
|
64
|
+
The key is built using the `identifier`. If you have a lot of arguments or
|
65
|
+
really long ones, you should consider overriding `identifier` to define a
|
66
|
+
more precise or loose custom identifier.
|
67
|
+
|
68
|
+
The default identifier is just your job arguments joined with a dash `-`.
|
69
|
+
|
70
|
+
By default the key uses this format:
|
71
|
+
`resque-lock-timeout:<job class name>:<identifier>`.
|
72
|
+
|
73
|
+
Or you can define the entire key by overriding `redis_lock_key`.
|
74
|
+
|
75
|
+
class UpdateNetworkGraph
|
76
|
+
extend Resque::Plugins::LockTimeout
|
77
|
+
@queue = :network_graph
|
78
|
+
|
79
|
+
# Run only one at a time, regardless of repo_id.
|
80
|
+
def self.identifier(repo_id)
|
81
|
+
nil
|
82
|
+
end
|
83
|
+
|
84
|
+
def self.perform(repo_id)
|
85
|
+
heavy_lifting
|
86
|
+
end
|
87
|
+
end
|
88
|
+
|
89
|
+
The above modification will ensure only one job of class
|
90
|
+
UpdateNetworkGraph is running at a time, regardless of the
|
91
|
+
repo_id.
|
92
|
+
|
93
|
+
It's lock key would be: `resque-lock-timeout:UpdateNetworkGraph`.
|
94
|
+
|
95
|
+
### Callbacks
|
96
|
+
|
97
|
+
Several callbacks are available to override and implement your own logic, e.g.
|
98
|
+
|
99
|
+
class UpdateNetworkGraph
|
100
|
+
extend Resque::Plugins::Lock
|
101
|
+
@queue = :network_graph
|
102
|
+
|
103
|
+
# Lock may be held for upto an hour.
|
104
|
+
@lock_timeout = 3600
|
105
|
+
|
106
|
+
# Job failed to acquire lock. You may implement retry or other logic.
|
107
|
+
def self.lock_failed(repo_id)
|
108
|
+
raise LockFailed
|
109
|
+
end
|
110
|
+
|
111
|
+
# Job has complete; but the lock expired before we could relase it.
|
112
|
+
# The lock wasn't released; as its *possible* the lock is now held
|
113
|
+
# by another job.
|
114
|
+
def self.lock_expired_before_release(repo_id)
|
115
|
+
handle_if_needed
|
116
|
+
end
|
117
|
+
|
118
|
+
def self.perform(repo_id)
|
119
|
+
heavy_lifting
|
120
|
+
end
|
121
|
+
end
|
122
|
+
|
123
|
+
Install
|
124
|
+
=======
|
125
|
+
|
126
|
+
$ gem install resque-lock-retry
|
127
|
+
|
128
|
+
Acknowledgements
|
129
|
+
================
|
130
|
+
|
131
|
+
Forked from Chris Wanstrath' [resque-lock][resque-lock] plugin.
|
132
|
+
Lock timeout from Ryan Carvar' [resque-lock-retry][resque-lock-retry] plugin.
|
133
|
+
And a little tinkering from Luke Antins.
|
134
|
+
|
135
|
+
[rq]: http://github.com/defunkt/resque
|
136
|
+
[redis-setnx]: http://code.google.com/p/redis/wiki/SetnxCommand
|
137
|
+
[resque-lock]: http://github.com/defunkt/resque-lock
|
138
|
+
[resque-lock-retry]: http://github.com/rcarver/resque-lock-retry
|
data/Rakefile
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
require 'rake/testtask'
|
2
|
+
require 'rake/rdoctask'
|
3
|
+
require 'yard'
|
4
|
+
require 'yard/rake/yardoc_task'
|
5
|
+
|
6
|
+
task :default => :test
|
7
|
+
|
8
|
+
desc 'Run tests.'
|
9
|
+
Rake::TestTask.new(:test) do |task|
|
10
|
+
task.test_files = FileList['test/*_test.rb']
|
11
|
+
task.verbose = true
|
12
|
+
end
|
13
|
+
|
14
|
+
desc 'Build Yardoc documentation.'
|
15
|
+
YARD::Rake::YardocTask.new :yardoc do |t|
|
16
|
+
t.files = ['lib/**/*.rb']
|
17
|
+
t.options = ['--output-dir', "doc/",
|
18
|
+
'--files', 'LICENSE',
|
19
|
+
'--readme', 'README.md',
|
20
|
+
'--title', 'resque-lock-timeout documentation']
|
21
|
+
end
|
@@ -0,0 +1 @@
|
|
1
|
+
require 'resque/plugins/lock_timeout'
|
@@ -0,0 +1,151 @@
|
|
1
|
+
module Resque
|
2
|
+
module Plugins
|
3
|
+
# If you want only one instance of your job running at a time,
|
4
|
+
# extend it with this module:
|
5
|
+
#
|
6
|
+
# require 'resque-lock'
|
7
|
+
#
|
8
|
+
# class UpdateNetworkGraph
|
9
|
+
# extend Resque::Plugins::LockTimeout
|
10
|
+
# @queue = :network_graph
|
11
|
+
#
|
12
|
+
# def self.perform(repo_id)
|
13
|
+
# heavy_lifting
|
14
|
+
# end
|
15
|
+
# end
|
16
|
+
#
|
17
|
+
# If you wish to limit the durati on a lock may be held for, you can
|
18
|
+
# set/override `lock_timeout`. e.g.
|
19
|
+
#
|
20
|
+
# class UpdateNetworkGraph
|
21
|
+
# extend Resque::Plugins::LockTimeout
|
22
|
+
# @queue = :network_graph
|
23
|
+
#
|
24
|
+
# # lock may be held for upto an hour.
|
25
|
+
# @lock_timeout = 3600
|
26
|
+
#
|
27
|
+
# def self.perform(repo_id)
|
28
|
+
# heavy_lifting
|
29
|
+
# end
|
30
|
+
# end
|
31
|
+
#
|
32
|
+
module LockTimeout
|
33
|
+
# @abstract You may override to implement a custom identifier,
|
34
|
+
# you should consider doing this if your job arguments
|
35
|
+
# are many/long or may not cleanly cleanly to strings.
|
36
|
+
#
|
37
|
+
# Builds an identifier using the job arguments. This identifier
|
38
|
+
# is used as part of the redis lock key.
|
39
|
+
#
|
40
|
+
# @param [Array] args job arguments
|
41
|
+
# @return [String, nil] job identifier
|
42
|
+
def identifier(*args)
|
43
|
+
args.join('-')
|
44
|
+
end
|
45
|
+
|
46
|
+
# Override to fully control the lock key used. It is passed
|
47
|
+
# the job arguments.
|
48
|
+
#
|
49
|
+
# The default looks like this:
|
50
|
+
# `resque-lock-timeout:<class name>:<identifier>`
|
51
|
+
#
|
52
|
+
# @return [String] redis key
|
53
|
+
def redis_lock_key(*args)
|
54
|
+
['lock', name, identifier(*args)].compact.join(":")
|
55
|
+
end
|
56
|
+
|
57
|
+
# Number of seconds the lock may be held for.
|
58
|
+
# A value of 0 or below will lock without a timeout.
|
59
|
+
#
|
60
|
+
# @return [Fixnum]
|
61
|
+
def lock_timeout
|
62
|
+
@lock_timeout ||= 0
|
63
|
+
end
|
64
|
+
|
65
|
+
# Try to acquire a lock.
|
66
|
+
#
|
67
|
+
# * Returns false; when unable to acquire the lock.
|
68
|
+
# * Returns true; when lock acquired, without a timeout.
|
69
|
+
# * Returns timestamp; when lock acquired with a timeout, timestamp is
|
70
|
+
# when the lock timeout expires.
|
71
|
+
#
|
72
|
+
# @return [Boolean, Fixnum]
|
73
|
+
def acquire_lock!(*args)
|
74
|
+
acquired = false
|
75
|
+
lock_key = redis_lock_key(*args)
|
76
|
+
|
77
|
+
unless lock_timeout > 0
|
78
|
+
# Acquire without using a timeout.
|
79
|
+
acquired = true if Resque.redis.setnx(lock_key, true)
|
80
|
+
else
|
81
|
+
# Acquire using the timeout algorithm.
|
82
|
+
acquired, lock_until = acquire_lock_algorithm!(lock_key)
|
83
|
+
end
|
84
|
+
|
85
|
+
lock_failed(*args) if !acquired && respond_to?(:lock_failed)
|
86
|
+
lock_until && acquired ? lock_until : acquired
|
87
|
+
end
|
88
|
+
|
89
|
+
# Attempts to aquire the lock using a timeout / deadlock algorithm.
|
90
|
+
#
|
91
|
+
# Locking algorithm: http://code.google.com/p/redis/wiki/SetnxCommand
|
92
|
+
def acquire_lock_algorithm!(lock_key)
|
93
|
+
now = Time.now.to_i
|
94
|
+
lock_until = now + lock_timeout
|
95
|
+
acquired = false
|
96
|
+
|
97
|
+
return [true, lock_until] if Resque.redis.setnx(lock_key, lock_until)
|
98
|
+
# Can't acquire the lock, see if it has expired.
|
99
|
+
lock_expiration = Resque.redis.get(lock_key)
|
100
|
+
if lock_expiration && lock_expiration.to_i < now
|
101
|
+
# expired, try to acquire.
|
102
|
+
lock_expiration = Resque.redis.getset(lock_key, lock_until)
|
103
|
+
if lock_expiration.nil? || lock_expiration.to_i < now
|
104
|
+
acquired = true
|
105
|
+
end
|
106
|
+
else
|
107
|
+
# Try once more...
|
108
|
+
acquired = true if Resque.redis.setnx(lock_key, lock_until)
|
109
|
+
end
|
110
|
+
|
111
|
+
[acquired, lock_until]
|
112
|
+
end
|
113
|
+
|
114
|
+
# Release the lock.
|
115
|
+
def release_lock!(*args)
|
116
|
+
Resque.redis.del(redis_lock_key(*args))
|
117
|
+
end
|
118
|
+
|
119
|
+
# Convenience method, not used internally.
|
120
|
+
#
|
121
|
+
# @return [Boolean] true if the job is locked by someone
|
122
|
+
def locked?(*args)
|
123
|
+
Resque.redis.exists(redis_lock_key(*args))
|
124
|
+
end
|
125
|
+
|
126
|
+
# Where the magic happens.
|
127
|
+
def around_perform_lock(*args)
|
128
|
+
# Abort if another job holds the lock.
|
129
|
+
return unless lock_until = acquire_lock!(*args)
|
130
|
+
|
131
|
+
begin
|
132
|
+
yield
|
133
|
+
ensure
|
134
|
+
# Release the lock on success and error. Unless a lock_timeout is
|
135
|
+
# used, then we need to be more careful before releasing the lock.
|
136
|
+
unless lock_until === true
|
137
|
+
now = Time.now.to_i
|
138
|
+
if lock_until < now && respond_to?(:lock_expired_before_release)
|
139
|
+
# Eeek! Lock expired before perform finished. Trigger callback.
|
140
|
+
lock_expired_before_release(*args)
|
141
|
+
return # dont relase lock.
|
142
|
+
end
|
143
|
+
end
|
144
|
+
release_lock!(*args)
|
145
|
+
end
|
146
|
+
end
|
147
|
+
|
148
|
+
end
|
149
|
+
|
150
|
+
end
|
151
|
+
end
|
data/test/lock_test.rb
ADDED
@@ -0,0 +1,108 @@
|
|
1
|
+
require File.dirname(__FILE__) + '/test_helper'
|
2
|
+
|
3
|
+
class LockTest < Test::Unit::TestCase
|
4
|
+
def setup
|
5
|
+
$success = $lock_failed = $lock_expired = 0
|
6
|
+
Resque.redis.flushall
|
7
|
+
@worker = Resque::Worker.new(:test)
|
8
|
+
end
|
9
|
+
|
10
|
+
def test_lint
|
11
|
+
assert_nothing_raised do
|
12
|
+
Resque::Plugin.lint(Resque::Plugins::LockTimeout)
|
13
|
+
end
|
14
|
+
end
|
15
|
+
|
16
|
+
def test_version
|
17
|
+
major, minor, patch = Resque::Version.split('.')
|
18
|
+
assert_equal 1, major.to_i
|
19
|
+
assert minor.to_i >= 7
|
20
|
+
end
|
21
|
+
|
22
|
+
def test_can_acquire_lock
|
23
|
+
SlowJob.acquire_lock!
|
24
|
+
assert_equal true, SlowJob.locked?, 'lock should be acquired'
|
25
|
+
end
|
26
|
+
|
27
|
+
def test_can_release_lock
|
28
|
+
SlowJob.acquire_lock!
|
29
|
+
assert_equal true, SlowJob.locked?, 'lock should be acquired'
|
30
|
+
|
31
|
+
SlowJob.release_lock!
|
32
|
+
assert_equal false, SlowJob.locked?, 'lock should have been released'
|
33
|
+
end
|
34
|
+
|
35
|
+
def test_lock_failed_callback
|
36
|
+
FastJob.acquire_lock!
|
37
|
+
assert_equal true, FastJob.locked?, 'lock should be acquired'
|
38
|
+
|
39
|
+
FastJob.acquire_lock!
|
40
|
+
assert_equal 1, $lock_failed, 'job callback should increment lock_failed'
|
41
|
+
end
|
42
|
+
|
43
|
+
def test_lock_without_timeout
|
44
|
+
3.times { Resque.enqueue(SlowJob) }
|
45
|
+
|
46
|
+
workers = []
|
47
|
+
3.times do
|
48
|
+
workers << Thread.new { @worker.process }
|
49
|
+
end
|
50
|
+
workers.each { |t| t.join }
|
51
|
+
|
52
|
+
assert_equal 1, $success, 'job should increment success'
|
53
|
+
end
|
54
|
+
|
55
|
+
def test_lock_is_released_on_success
|
56
|
+
Resque.enqueue(FastJob)
|
57
|
+
@worker.process
|
58
|
+
assert_equal 1, $success, 'job should increment success'
|
59
|
+
assert_equal false, FastJob.locked?, 'lock should have been released'
|
60
|
+
end
|
61
|
+
|
62
|
+
def test_lock_is_released_on_failure
|
63
|
+
Resque.enqueue(FailingFastJob)
|
64
|
+
@worker.process
|
65
|
+
assert_equal 0, $success, 'job shouldnt increment success'
|
66
|
+
assert_equal false, FastJob.locked?, 'lock should have been released'
|
67
|
+
end
|
68
|
+
|
69
|
+
def test_can_acquire_lock_with_timeout
|
70
|
+
now = Time.now.to_i
|
71
|
+
assert SlowWithTimeoutJob.acquire_lock!, 'acquire lock'
|
72
|
+
|
73
|
+
lock = Resque.redis.get(SlowWithTimeoutJob.redis_lock_key)
|
74
|
+
assert (now + 58) < lock.to_i, 'lock expire time should be in the future'
|
75
|
+
end
|
76
|
+
|
77
|
+
def test_lock_recovers_after_lock_timeout
|
78
|
+
now = Time.now.to_i
|
79
|
+
assert SlowWithTimeoutJob.acquire_lock!, 'acquire lock'
|
80
|
+
assert_equal false, SlowWithTimeoutJob.acquire_lock!, 'acquire lock fails'
|
81
|
+
|
82
|
+
Resque.redis.set(SlowWithTimeoutJob.redis_lock_key, now - 40) # haxor timeout.
|
83
|
+
assert SlowWithTimeoutJob.acquire_lock!, 'acquire lock, timeout expired'
|
84
|
+
|
85
|
+
lock = Resque.redis.get(SlowWithTimeoutJob.redis_lock_key)
|
86
|
+
assert (now + 58) < lock.to_i
|
87
|
+
end
|
88
|
+
|
89
|
+
def test_lock_with_timeout
|
90
|
+
3.times { Resque.enqueue(SlowWithTimeoutJob) }
|
91
|
+
|
92
|
+
workers = []
|
93
|
+
3.times do
|
94
|
+
workers << Thread.new { @worker.process }
|
95
|
+
end
|
96
|
+
workers.each { |t| t.join }
|
97
|
+
|
98
|
+
assert_equal 1, $success, 'job should increment success'
|
99
|
+
end
|
100
|
+
|
101
|
+
def test_lock_expired_before_release
|
102
|
+
Resque.enqueue(ExpireBeforeReleaseJob)
|
103
|
+
@worker.process
|
104
|
+
assert_equal 1, $success, 'job should increment success'
|
105
|
+
assert_equal true, $lock_expired, 'should be set by callback method'
|
106
|
+
assert_equal false, FastJob.locked?, 'lock should not release'
|
107
|
+
end
|
108
|
+
end
|
@@ -0,0 +1,132 @@
|
|
1
|
+
# Redis configuration file example
|
2
|
+
|
3
|
+
# By default Redis does not run as a daemon. Use 'yes' if you need it.
|
4
|
+
# Note that Redis will write a pid file in /var/run/redis.pid when daemonized.
|
5
|
+
daemonize yes
|
6
|
+
|
7
|
+
# When run as a daemon, Redis write a pid file in /var/run/redis.pid by default.
|
8
|
+
# You can specify a custom pid file location here.
|
9
|
+
pidfile ./test/redis-test.pid
|
10
|
+
|
11
|
+
# Accept connections on the specified port, default is 6379
|
12
|
+
port 9736
|
13
|
+
|
14
|
+
# If you want you can bind a single interface, if the bind option is not
|
15
|
+
# specified all the interfaces will listen for connections.
|
16
|
+
#
|
17
|
+
# bind 127.0.0.1
|
18
|
+
|
19
|
+
# Close the connection after a client is idle for N seconds (0 to disable)
|
20
|
+
timeout 300
|
21
|
+
|
22
|
+
# Save the DB on disk:
|
23
|
+
#
|
24
|
+
# save <seconds> <changes>
|
25
|
+
#
|
26
|
+
# Will save the DB if both the given number of seconds and the given
|
27
|
+
# number of write operations against the DB occurred.
|
28
|
+
#
|
29
|
+
# In the example below the behaviour will be to save:
|
30
|
+
# after 900 sec (15 min) if at least 1 key changed
|
31
|
+
# after 300 sec (5 min) if at least 10 keys changed
|
32
|
+
# after 60 sec if at least 10000 keys changed
|
33
|
+
save 900 1
|
34
|
+
save 300 10
|
35
|
+
save 60 10000
|
36
|
+
|
37
|
+
# The filename where to dump the DB
|
38
|
+
dbfilename dump.rdb
|
39
|
+
|
40
|
+
# For default save/load DB in/from the working directory
|
41
|
+
# Note that you must specify a directory not a file name.
|
42
|
+
dir ./test/
|
43
|
+
|
44
|
+
# Set server verbosity to 'debug'
|
45
|
+
# it can be one of:
|
46
|
+
# debug (a lot of information, useful for development/testing)
|
47
|
+
# notice (moderately verbose, what you want in production probably)
|
48
|
+
# warning (only very important / critical messages are logged)
|
49
|
+
loglevel debug
|
50
|
+
|
51
|
+
# Specify the log file name. Also 'stdout' can be used to force
|
52
|
+
# the demon to log on the standard output. Note that if you use standard
|
53
|
+
# output for logging but daemonize, logs will be sent to /dev/null
|
54
|
+
logfile stdout
|
55
|
+
|
56
|
+
# Set the number of databases. The default database is DB 0, you can select
|
57
|
+
# a different one on a per-connection basis using SELECT <dbid> where
|
58
|
+
# dbid is a number between 0 and 'databases'-1
|
59
|
+
databases 16
|
60
|
+
|
61
|
+
################################# REPLICATION #################################
|
62
|
+
|
63
|
+
# Master-Slave replication. Use slaveof to make a Redis instance a copy of
|
64
|
+
# another Redis server. Note that the configuration is local to the slave
|
65
|
+
# so for example it is possible to configure the slave to save the DB with a
|
66
|
+
# different interval, or to listen to another port, and so on.
|
67
|
+
|
68
|
+
# slaveof <masterip> <masterport>
|
69
|
+
|
70
|
+
################################## SECURITY ###################################
|
71
|
+
|
72
|
+
# Require clients to issue AUTH <PASSWORD> before processing any other
|
73
|
+
# commands. This might be useful in environments in which you do not trust
|
74
|
+
# others with access to the host running redis-server.
|
75
|
+
#
|
76
|
+
# This should stay commented out for backward compatibility and because most
|
77
|
+
# people do not need auth (e.g. they run their own servers).
|
78
|
+
|
79
|
+
# requirepass foobared
|
80
|
+
|
81
|
+
################################### LIMITS ####################################
|
82
|
+
|
83
|
+
# Set the max number of connected clients at the same time. By default there
|
84
|
+
# is no limit, and it's up to the number of file descriptors the Redis process
|
85
|
+
# is able to open. The special value '0' means no limts.
|
86
|
+
# Once the limit is reached Redis will close all the new connections sending
|
87
|
+
# an error 'max number of clients reached'.
|
88
|
+
|
89
|
+
# maxclients 128
|
90
|
+
|
91
|
+
# Don't use more memory than the specified amount of bytes.
|
92
|
+
# When the memory limit is reached Redis will try to remove keys with an
|
93
|
+
# EXPIRE set. It will try to start freeing keys that are going to expire
|
94
|
+
# in little time and preserve keys with a longer time to live.
|
95
|
+
# Redis will also try to remove objects from free lists if possible.
|
96
|
+
#
|
97
|
+
# If all this fails, Redis will start to reply with errors to commands
|
98
|
+
# that will use more memory, like SET, LPUSH, and so on, and will continue
|
99
|
+
# to reply to most read-only commands like GET.
|
100
|
+
#
|
101
|
+
# WARNING: maxmemory can be a good idea mainly if you want to use Redis as a
|
102
|
+
# 'state' server or cache, not as a real DB. When Redis is used as a real
|
103
|
+
# database the memory usage will grow over the weeks, it will be obvious if
|
104
|
+
# it is going to use too much memory in the long run, and you'll have the time
|
105
|
+
# to upgrade. With maxmemory after the limit is reached you'll start to get
|
106
|
+
# errors for write operations, and this may even lead to DB inconsistency.
|
107
|
+
|
108
|
+
# maxmemory <bytes>
|
109
|
+
|
110
|
+
############################### ADVANCED CONFIG ###############################
|
111
|
+
|
112
|
+
# Glue small output buffers together in order to send small replies in a
|
113
|
+
# single TCP packet. Uses a bit more CPU but most of the times it is a win
|
114
|
+
# in terms of number of queries per second. Use 'yes' if unsure.
|
115
|
+
glueoutputbuf yes
|
116
|
+
|
117
|
+
# Use object sharing. Can save a lot of memory if you have many common
|
118
|
+
# string in your dataset, but performs lookups against the shared objects
|
119
|
+
# pool so it uses more CPU and can be a bit slower. Usually it's a good
|
120
|
+
# idea.
|
121
|
+
#
|
122
|
+
# When object sharing is enabled (shareobjects yes) you can use
|
123
|
+
# shareobjectspoolsize to control the size of the pool used in order to try
|
124
|
+
# object sharing. A bigger pool size will lead to better sharing capabilities.
|
125
|
+
# In general you want this value to be at least the double of the number of
|
126
|
+
# very common strings you have in your dataset.
|
127
|
+
#
|
128
|
+
# WARNING: object sharing is experimental, don't enable this feature
|
129
|
+
# in production before of Redis 1.0-stable. Still please try this feature in
|
130
|
+
# your development environment so that we can test it better.
|
131
|
+
shareobjects no
|
132
|
+
shareobjectspoolsize 1024
|
data/test/test_helper.rb
ADDED
@@ -0,0 +1,41 @@
|
|
1
|
+
dir = File.dirname(File.expand_path(__FILE__))
|
2
|
+
$LOAD_PATH.unshift dir + '/../lib'
|
3
|
+
$TESTING = true
|
4
|
+
|
5
|
+
require 'test/unit'
|
6
|
+
require 'resque'
|
7
|
+
require 'turn'
|
8
|
+
|
9
|
+
require 'resque-lock-timeout'
|
10
|
+
require dir + '/test_jobs'
|
11
|
+
|
12
|
+
##
|
13
|
+
# make sure we can run redis
|
14
|
+
if !system("which redis-server")
|
15
|
+
puts '', "** can't find `redis-server` in your path"
|
16
|
+
puts "** try running `sudo rake install`"
|
17
|
+
abort ''
|
18
|
+
end
|
19
|
+
|
20
|
+
##
|
21
|
+
# start our own redis when the tests start,
|
22
|
+
# kill it when they end
|
23
|
+
at_exit do
|
24
|
+
next if $!
|
25
|
+
|
26
|
+
if defined?(MiniTest)
|
27
|
+
exit_code = MiniTest::Unit.new.run(ARGV)
|
28
|
+
else
|
29
|
+
exit_code = Test::Unit::AutoRunner.run
|
30
|
+
end
|
31
|
+
|
32
|
+
pid = `ps -e -o pid,command | grep [r]edis-test`.split(" ")[0]
|
33
|
+
puts "Killing test redis server..."
|
34
|
+
`rm -f #{dir}/dump.rdb`
|
35
|
+
Process.kill("KILL", pid.to_i)
|
36
|
+
exit exit_code
|
37
|
+
end
|
38
|
+
|
39
|
+
puts "Starting redis for testing at localhost:9736..."
|
40
|
+
`redis-server #{dir}/redis-test.conf`
|
41
|
+
Resque.redis = '127.0.0.1:9736'
|
data/test/test_jobs.rb
ADDED
@@ -0,0 +1,62 @@
|
|
1
|
+
class SlowJob
|
2
|
+
extend Resque::Plugins::LockTimeout
|
3
|
+
@queue = :test
|
4
|
+
|
5
|
+
def self.perform
|
6
|
+
$success += 1
|
7
|
+
sleep 0.2
|
8
|
+
end
|
9
|
+
|
10
|
+
def self.lock_failed(*args)
|
11
|
+
$lock_failed += 1
|
12
|
+
end
|
13
|
+
end
|
14
|
+
|
15
|
+
class FastJob
|
16
|
+
extend Resque::Plugins::LockTimeout
|
17
|
+
@queue = :test
|
18
|
+
|
19
|
+
def self.perform
|
20
|
+
$success += 1
|
21
|
+
end
|
22
|
+
|
23
|
+
def self.lock_failed(*args)
|
24
|
+
$lock_failed += 1
|
25
|
+
end
|
26
|
+
end
|
27
|
+
|
28
|
+
class FailingFastJob
|
29
|
+
extend Resque::Plugins::LockTimeout
|
30
|
+
@queue = :test
|
31
|
+
|
32
|
+
def self.perform
|
33
|
+
raise
|
34
|
+
$success += 1
|
35
|
+
end
|
36
|
+
end
|
37
|
+
|
38
|
+
class SlowWithTimeoutJob
|
39
|
+
extend Resque::Plugins::LockTimeout
|
40
|
+
@queue = :test
|
41
|
+
@lock_timeout = 60
|
42
|
+
|
43
|
+
def self.perform
|
44
|
+
$success += 1
|
45
|
+
sleep 0.2
|
46
|
+
end
|
47
|
+
end
|
48
|
+
|
49
|
+
class ExpireBeforeReleaseJob
|
50
|
+
extend Resque::Plugins::LockTimeout
|
51
|
+
@queue = :test
|
52
|
+
@lock_timeout = 1
|
53
|
+
|
54
|
+
def self.perform
|
55
|
+
$success += 1
|
56
|
+
sleep 2
|
57
|
+
end
|
58
|
+
|
59
|
+
def self.lock_expired_before_release
|
60
|
+
$lock_expired = true
|
61
|
+
end
|
62
|
+
end
|
metadata
ADDED
@@ -0,0 +1,109 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: resque-lock-timeout
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
prerelease: false
|
5
|
+
segments:
|
6
|
+
- 0
|
7
|
+
- 2
|
8
|
+
- 0
|
9
|
+
version: 0.2.0
|
10
|
+
platform: ruby
|
11
|
+
authors:
|
12
|
+
- Luke Antins
|
13
|
+
- Ryan Carver
|
14
|
+
- Chris Wanstrath
|
15
|
+
autorequire:
|
16
|
+
bindir: bin
|
17
|
+
cert_chain: []
|
18
|
+
|
19
|
+
date: 2010-05-06 00:00:00 +01:00
|
20
|
+
default_executable:
|
21
|
+
dependencies:
|
22
|
+
- !ruby/object:Gem::Dependency
|
23
|
+
name: resque
|
24
|
+
prerelease: false
|
25
|
+
requirement: &id001 !ruby/object:Gem::Requirement
|
26
|
+
requirements:
|
27
|
+
- - ~>
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
segments:
|
30
|
+
- 1
|
31
|
+
- 8
|
32
|
+
- 0
|
33
|
+
version: 1.8.0
|
34
|
+
type: :runtime
|
35
|
+
version_requirements: *id001
|
36
|
+
- !ruby/object:Gem::Dependency
|
37
|
+
name: turn
|
38
|
+
prerelease: false
|
39
|
+
requirement: &id002 !ruby/object:Gem::Requirement
|
40
|
+
requirements:
|
41
|
+
- - ">="
|
42
|
+
- !ruby/object:Gem::Version
|
43
|
+
segments:
|
44
|
+
- 0
|
45
|
+
version: "0"
|
46
|
+
type: :development
|
47
|
+
version_requirements: *id002
|
48
|
+
- !ruby/object:Gem::Dependency
|
49
|
+
name: yard
|
50
|
+
prerelease: false
|
51
|
+
requirement: &id003 !ruby/object:Gem::Requirement
|
52
|
+
requirements:
|
53
|
+
- - ">="
|
54
|
+
- !ruby/object:Gem::Version
|
55
|
+
segments:
|
56
|
+
- 0
|
57
|
+
version: "0"
|
58
|
+
type: :development
|
59
|
+
version_requirements: *id003
|
60
|
+
description: " A Resque plugin. Adds locking, with optional timeout/deadlock handling to\n resque jobs.\n\n Using a `lock_timeout` allows you to re-aquire the lock should your worker\n fail, crash, or is otherwise unable to relase the lock.\n \n i.e. Your server unexpectedly looses power. Very handy for jobs that are\n recurring or may be retried.\n"
|
61
|
+
email: luke@lividpenguin.com
|
62
|
+
executables: []
|
63
|
+
|
64
|
+
extensions: []
|
65
|
+
|
66
|
+
extra_rdoc_files: []
|
67
|
+
|
68
|
+
files:
|
69
|
+
- README.md
|
70
|
+
- Rakefile
|
71
|
+
- LICENSE
|
72
|
+
- lib/resque/plugins/lock_timeout.rb
|
73
|
+
- lib/resque-lock-timeout.rb
|
74
|
+
- test/lock_test.rb
|
75
|
+
- test/redis-test.conf
|
76
|
+
- test/test_helper.rb
|
77
|
+
- test/test_jobs.rb
|
78
|
+
has_rdoc: false
|
79
|
+
homepage: http://github.com/lantins/resque-lock-timeout
|
80
|
+
licenses: []
|
81
|
+
|
82
|
+
post_install_message:
|
83
|
+
rdoc_options: []
|
84
|
+
|
85
|
+
require_paths:
|
86
|
+
- lib
|
87
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
88
|
+
requirements:
|
89
|
+
- - ">="
|
90
|
+
- !ruby/object:Gem::Version
|
91
|
+
segments:
|
92
|
+
- 0
|
93
|
+
version: "0"
|
94
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
95
|
+
requirements:
|
96
|
+
- - ">="
|
97
|
+
- !ruby/object:Gem::Version
|
98
|
+
segments:
|
99
|
+
- 0
|
100
|
+
version: "0"
|
101
|
+
requirements: []
|
102
|
+
|
103
|
+
rubyforge_project:
|
104
|
+
rubygems_version: 1.3.6
|
105
|
+
signing_key:
|
106
|
+
specification_version: 3
|
107
|
+
summary: A Resque plugin adding locking, with optional timeout/deadlock handling to resque jobs.
|
108
|
+
test_files: []
|
109
|
+
|