resque-uniqueue 0.2.2 → 0.3.0
Sign up to get free protection for your applications and to get access to all the features.
- data/Gemfile +2 -2
- data/Gemfile.lock +10 -10
- data/README.md +70 -0
- data/VERSION +1 -1
- data/lib/resque/uniqueue.rb +8 -8
- data/resque-uniqueue.gemspec +10 -10
- metadata +9 -9
- data/README.rdoc +0 -19
data/Gemfile
CHANGED
@@ -5,13 +5,13 @@ source "http://rubygems.org"
|
|
5
5
|
|
6
6
|
# Add dependencies to develop your gem here.
|
7
7
|
# Include everything needed to run rake, tests, features, etc.
|
8
|
-
gem 'resque', '~> 1.
|
8
|
+
gem 'resque', '~> 1.25.0'
|
9
9
|
|
10
10
|
group :development do
|
11
11
|
gem 'mocha'
|
12
12
|
gem "shoulda", ">= 0"
|
13
13
|
gem "rdoc", "~> 3.12"
|
14
|
-
gem "bundler", "~> 1.
|
14
|
+
gem "bundler", "~> 1.5.0"
|
15
15
|
gem "jeweler", "~> 1.8.4"
|
16
16
|
gem "pry"
|
17
17
|
end
|
data/Gemfile.lock
CHANGED
@@ -24,18 +24,18 @@ GEM
|
|
24
24
|
method_source (~> 0.8)
|
25
25
|
slop (~> 3.4)
|
26
26
|
rack (1.5.2)
|
27
|
-
rack-protection (1.5.
|
27
|
+
rack-protection (1.5.3)
|
28
28
|
rack
|
29
29
|
rake (10.0.4)
|
30
30
|
rdoc (3.12.2)
|
31
31
|
json (~> 1.4)
|
32
|
-
redis (3.0.
|
33
|
-
redis-namespace (1.
|
34
|
-
redis (~> 3.0.
|
35
|
-
resque (1.
|
32
|
+
redis (3.0.7)
|
33
|
+
redis-namespace (1.4.1)
|
34
|
+
redis (~> 3.0.4)
|
35
|
+
resque (1.25.2)
|
36
36
|
mono_logger (~> 1.0)
|
37
37
|
multi_json (~> 1.0)
|
38
|
-
redis-namespace (~> 1.
|
38
|
+
redis-namespace (~> 1.3)
|
39
39
|
sinatra (>= 0.9.2)
|
40
40
|
vegas (~> 0.1.2)
|
41
41
|
shoulda (3.5.0)
|
@@ -44,8 +44,8 @@ GEM
|
|
44
44
|
shoulda-context (1.1.2)
|
45
45
|
shoulda-matchers (2.1.0)
|
46
46
|
activesupport (>= 3.0.0)
|
47
|
-
sinatra (1.4.
|
48
|
-
rack (~> 1.
|
47
|
+
sinatra (1.4.5)
|
48
|
+
rack (~> 1.4)
|
49
49
|
rack-protection (~> 1.4)
|
50
50
|
tilt (~> 1.3, >= 1.3.4)
|
51
51
|
slop (3.4.3)
|
@@ -57,10 +57,10 @@ PLATFORMS
|
|
57
57
|
ruby
|
58
58
|
|
59
59
|
DEPENDENCIES
|
60
|
-
bundler (~> 1.
|
60
|
+
bundler (~> 1.5.0)
|
61
61
|
jeweler (~> 1.8.4)
|
62
62
|
mocha
|
63
63
|
pry
|
64
64
|
rdoc (~> 3.12)
|
65
|
-
resque (~> 1.
|
65
|
+
resque (~> 1.25.0)
|
66
66
|
shoulda
|
data/README.md
ADDED
@@ -0,0 +1,70 @@
|
|
1
|
+
# resque-uniqueue
|
2
|
+
|
3
|
+
### Why use Uniqueue?
|
4
|
+
|
5
|
+
Multiple identical jobs are wasteful. Uniqueue ensures that *for a given queue*, identical jobs are never enqueued, and you can stress a little less about your application code causing duplicate work.
|
6
|
+
|
7
|
+
### Prerequisites
|
8
|
+
Uniqueue uses Lua scripting, which requires **Redis 2.6 or greater**, so make sure your Redis installation is up to date.
|
9
|
+
|
10
|
+
Before deploying an application with Uniqueue enabled for the first time, it's important that you **ensure your queues are empty**.
|
11
|
+
|
12
|
+
### Installation
|
13
|
+
Add Uniqueue to your Gemfile.
|
14
|
+
|
15
|
+
gem 'resque-uniqueue'
|
16
|
+
|
17
|
+
Then run bundle install within your app's directory.
|
18
|
+
|
19
|
+
### Configuration
|
20
|
+
You'll need to configure Uniqueue somewhere in your application's initialization process. If you are running a Rails application it's recommended you use your Resque initializer:
|
21
|
+
|
22
|
+
# config/initializers/resque.rb
|
23
|
+
Resque.unique_queues!
|
24
|
+
|
25
|
+
#### Specifying queues
|
26
|
+
By default Uniqueue defaults to preventing identical jobs on all queues. However, if you need to scope Uniqueue to specific queues, then your intializer code should look like this:
|
27
|
+
|
28
|
+
# config/initializers/resque.rb
|
29
|
+
Resque.unique_queues = ["emails", "orders"]
|
30
|
+
Resque.unique_queues!
|
31
|
+
|
32
|
+
### How It Works
|
33
|
+
|
34
|
+
Uniqueue overrides 3 resque commands: `push`, `pop`, and `remove_queue` in order to enforce *queue-level uniqueness* of jobs. And for each queue two additional Redis keys are created:
|
35
|
+
|
36
|
+
1. `queue:[queue_name]:uniqueue` - A **set** containing MultiJSON dumps of the payload of all items on the queue
|
37
|
+
2. `queue:[queue_name]:start_at` - A **list** containing the Unix timestamp of each job on the queue's start time, ordered identically to the actual job queue
|
38
|
+
|
39
|
+
Now when Resque pushes a job the following happens:
|
40
|
+
|
41
|
+
1. The length/cardinality of the queue, uniqueue set, and start_at list are verified to be equal. If they aren't, stuff has gone bad, and you'll get an exception.
|
42
|
+
2. A Lua script is evaluated that executes `sadd` on the payload (well, a MultiJSON dump of it), which will add it to the uniqueue set if it is not already a member.
|
43
|
+
3. If the payload's dump was not previously stored in the set, we `rpush` the start time of the job to the start_at list and `rpush` the job to the queue (following the lead of Resque's default `push` command).
|
44
|
+
|
45
|
+
Because the three operations happen in the context of a Lua script, atomicity is guaranteed (See "Atomicity of Scripts" [here][eval]), and race conditions can never cause the uniqueue set, start_at list, and original queue to get out of sync. Each unique job is now successfully queued, its payload is present in our uniqueue set, and its start_at timestamp is on a list that corresponds exactly to the order of the queue list.
|
46
|
+
|
47
|
+
Popping a job is very similar:
|
48
|
+
|
49
|
+
1. Lengths and cardinalities are verified.
|
50
|
+
2. MultiJSON load the payload.
|
51
|
+
3. Return the payload with an additional key of 'start_at', which is the Unix timestamp that the job began.
|
52
|
+
|
53
|
+
And this is how a unique job is born.
|
54
|
+
|
55
|
+
### Contributing
|
56
|
+
|
57
|
+
* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.
|
58
|
+
* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it.
|
59
|
+
* Fork the project.
|
60
|
+
* Start a feature/bugfix branch.
|
61
|
+
* Commit and push until you are happy with your contribution.
|
62
|
+
* Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
|
63
|
+
* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
|
64
|
+
|
65
|
+
### Copyright
|
66
|
+
|
67
|
+
Copyright (c) 2013 AcademicWorks, inc. See LICENSE.txt for
|
68
|
+
further details.
|
69
|
+
|
70
|
+
[eval]: http://redis.io/commands/eval
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
0.
|
1
|
+
0.3.0
|
data/lib/resque/uniqueue.rb
CHANGED
@@ -1,12 +1,12 @@
|
|
1
|
-
require
|
1
|
+
require "resque"
|
2
2
|
|
3
3
|
module Resque
|
4
4
|
module Uniqueue
|
5
|
-
|
5
|
+
|
6
6
|
def push(queue, item)
|
7
7
|
unique_queue?(queue) ? push_unique(queue, item) : super
|
8
8
|
end
|
9
|
-
|
9
|
+
|
10
10
|
def pop(queue)
|
11
11
|
unique_queue?(queue) ? pop_unique(queue) : super
|
12
12
|
end
|
@@ -30,7 +30,7 @@ module Resque
|
|
30
30
|
results = redis.evalsha pop_unique_eval_sha, [queue]
|
31
31
|
return nil unless results[0]
|
32
32
|
job = decode results[0]
|
33
|
-
job[
|
33
|
+
job["start_at"] ||= results[1].to_i
|
34
34
|
return job
|
35
35
|
end
|
36
36
|
|
@@ -39,8 +39,8 @@ module Resque
|
|
39
39
|
local queue_name = KEYS[1]
|
40
40
|
local uniqueue_name = queue_name..':uniqueue'
|
41
41
|
local start_at_name = queue_name..':start_at'
|
42
|
-
local
|
43
|
-
if
|
42
|
+
local not_in_set = redis.call('sadd', uniqueue_name , ARGV[1])
|
43
|
+
if not_in_set == 1 then
|
44
44
|
redis.call('rpush', start_at_name, ARGV[2])
|
45
45
|
return redis.call('rpush', queue_name, ARGV[1])
|
46
46
|
end
|
@@ -84,7 +84,7 @@ module Resque
|
|
84
84
|
|
85
85
|
#if the queue and set sizes differ, something is very wrong and we should fail loudly
|
86
86
|
def confirm_unique_queue_validity(queue)
|
87
|
-
response =
|
87
|
+
response = redis.evalsha queue_and_set_length_equal_eval_sha, [queue]
|
88
88
|
return true if response == 1
|
89
89
|
#TODO raise specific exception
|
90
90
|
raise "Make sure your queues are empty before you start using uniqueue"
|
@@ -120,7 +120,7 @@ module Resque
|
|
120
120
|
|
121
121
|
def confirm_compatible_redis_version
|
122
122
|
redis_version = redis.info["redis_version"]
|
123
|
-
major, minor, patch = redis_version.split(
|
123
|
+
major, minor, patch = redis_version.split(".").map(&:to_i)
|
124
124
|
if major < 2 || minor < 6
|
125
125
|
#TODO raise specific exception
|
126
126
|
raise "Redis version must be at least 2.6.0 you are running #{redis_version}"
|
data/resque-uniqueue.gemspec
CHANGED
@@ -5,23 +5,23 @@
|
|
5
5
|
|
6
6
|
Gem::Specification.new do |s|
|
7
7
|
s.name = "resque-uniqueue"
|
8
|
-
s.version = "0.
|
8
|
+
s.version = "0.3.0"
|
9
9
|
|
10
10
|
s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
|
11
11
|
s.authors = ["Aaron Scruggs"]
|
12
|
-
s.date = "
|
12
|
+
s.date = "2014-04-20"
|
13
13
|
s.description = "Unique Resque queues using redis 1.6.0 scripting, sets and not much else"
|
14
14
|
s.email = "aaron@scrug.gs"
|
15
15
|
s.extra_rdoc_files = [
|
16
16
|
"LICENSE.txt",
|
17
|
-
"README.
|
17
|
+
"README.md"
|
18
18
|
]
|
19
19
|
s.files = [
|
20
20
|
".document",
|
21
21
|
"Gemfile",
|
22
22
|
"Gemfile.lock",
|
23
23
|
"LICENSE.txt",
|
24
|
-
"README.
|
24
|
+
"README.md",
|
25
25
|
"Rakefile",
|
26
26
|
"VERSION",
|
27
27
|
"lib/resque-uniqueue.rb",
|
@@ -40,28 +40,28 @@ Gem::Specification.new do |s|
|
|
40
40
|
s.specification_version = 3
|
41
41
|
|
42
42
|
if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
|
43
|
-
s.add_runtime_dependency(%q<resque>, ["~> 1.
|
43
|
+
s.add_runtime_dependency(%q<resque>, ["~> 1.25.0"])
|
44
44
|
s.add_development_dependency(%q<mocha>, [">= 0"])
|
45
45
|
s.add_development_dependency(%q<shoulda>, [">= 0"])
|
46
46
|
s.add_development_dependency(%q<rdoc>, ["~> 3.12"])
|
47
|
-
s.add_development_dependency(%q<bundler>, ["~> 1.
|
47
|
+
s.add_development_dependency(%q<bundler>, ["~> 1.5.0"])
|
48
48
|
s.add_development_dependency(%q<jeweler>, ["~> 1.8.4"])
|
49
49
|
s.add_development_dependency(%q<pry>, [">= 0"])
|
50
50
|
else
|
51
|
-
s.add_dependency(%q<resque>, ["~> 1.
|
51
|
+
s.add_dependency(%q<resque>, ["~> 1.25.0"])
|
52
52
|
s.add_dependency(%q<mocha>, [">= 0"])
|
53
53
|
s.add_dependency(%q<shoulda>, [">= 0"])
|
54
54
|
s.add_dependency(%q<rdoc>, ["~> 3.12"])
|
55
|
-
s.add_dependency(%q<bundler>, ["~> 1.
|
55
|
+
s.add_dependency(%q<bundler>, ["~> 1.5.0"])
|
56
56
|
s.add_dependency(%q<jeweler>, ["~> 1.8.4"])
|
57
57
|
s.add_dependency(%q<pry>, [">= 0"])
|
58
58
|
end
|
59
59
|
else
|
60
|
-
s.add_dependency(%q<resque>, ["~> 1.
|
60
|
+
s.add_dependency(%q<resque>, ["~> 1.25.0"])
|
61
61
|
s.add_dependency(%q<mocha>, [">= 0"])
|
62
62
|
s.add_dependency(%q<shoulda>, [">= 0"])
|
63
63
|
s.add_dependency(%q<rdoc>, ["~> 3.12"])
|
64
|
-
s.add_dependency(%q<bundler>, ["~> 1.
|
64
|
+
s.add_dependency(%q<bundler>, ["~> 1.5.0"])
|
65
65
|
s.add_dependency(%q<jeweler>, ["~> 1.8.4"])
|
66
66
|
s.add_dependency(%q<pry>, [">= 0"])
|
67
67
|
end
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: resque-uniqueue
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.3.0
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date:
|
12
|
+
date: 2014-04-20 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: resque
|
@@ -18,7 +18,7 @@ dependencies:
|
|
18
18
|
requirements:
|
19
19
|
- - ~>
|
20
20
|
- !ruby/object:Gem::Version
|
21
|
-
version: 1.
|
21
|
+
version: 1.25.0
|
22
22
|
type: :runtime
|
23
23
|
prerelease: false
|
24
24
|
version_requirements: !ruby/object:Gem::Requirement
|
@@ -26,7 +26,7 @@ dependencies:
|
|
26
26
|
requirements:
|
27
27
|
- - ~>
|
28
28
|
- !ruby/object:Gem::Version
|
29
|
-
version: 1.
|
29
|
+
version: 1.25.0
|
30
30
|
- !ruby/object:Gem::Dependency
|
31
31
|
name: mocha
|
32
32
|
requirement: !ruby/object:Gem::Requirement
|
@@ -82,7 +82,7 @@ dependencies:
|
|
82
82
|
requirements:
|
83
83
|
- - ~>
|
84
84
|
- !ruby/object:Gem::Version
|
85
|
-
version: 1.
|
85
|
+
version: 1.5.0
|
86
86
|
type: :development
|
87
87
|
prerelease: false
|
88
88
|
version_requirements: !ruby/object:Gem::Requirement
|
@@ -90,7 +90,7 @@ dependencies:
|
|
90
90
|
requirements:
|
91
91
|
- - ~>
|
92
92
|
- !ruby/object:Gem::Version
|
93
|
-
version: 1.
|
93
|
+
version: 1.5.0
|
94
94
|
- !ruby/object:Gem::Dependency
|
95
95
|
name: jeweler
|
96
96
|
requirement: !ruby/object:Gem::Requirement
|
@@ -129,13 +129,13 @@ executables: []
|
|
129
129
|
extensions: []
|
130
130
|
extra_rdoc_files:
|
131
131
|
- LICENSE.txt
|
132
|
-
- README.
|
132
|
+
- README.md
|
133
133
|
files:
|
134
134
|
- .document
|
135
135
|
- Gemfile
|
136
136
|
- Gemfile.lock
|
137
137
|
- LICENSE.txt
|
138
|
-
- README.
|
138
|
+
- README.md
|
139
139
|
- Rakefile
|
140
140
|
- VERSION
|
141
141
|
- lib/resque-uniqueue.rb
|
@@ -158,7 +158,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
158
158
|
version: '0'
|
159
159
|
segments:
|
160
160
|
- 0
|
161
|
-
hash: -
|
161
|
+
hash: -2465876499125034208
|
162
162
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
163
163
|
none: false
|
164
164
|
requirements:
|
data/README.rdoc
DELETED
@@ -1,19 +0,0 @@
|
|
1
|
-
= resque-uniqueue
|
2
|
-
|
3
|
-
Description goes here.
|
4
|
-
|
5
|
-
== Contributing to resque-uniqueue
|
6
|
-
|
7
|
-
* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.
|
8
|
-
* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it.
|
9
|
-
* Fork the project.
|
10
|
-
* Start a feature/bugfix branch.
|
11
|
-
* Commit and push until you are happy with your contribution.
|
12
|
-
* Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
|
13
|
-
* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
|
14
|
-
|
15
|
-
== Copyright
|
16
|
-
|
17
|
-
Copyright (c) 2013 AcademicWorks, inc. See LICENSE.txt for
|
18
|
-
further details.
|
19
|
-
|