qu 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- data/.gitignore +4 -0
- data/Gemfile +19 -0
- data/Guardfile +12 -0
- data/README.md +140 -0
- data/Rakefile +46 -0
- data/lib/qu.rb +27 -0
- data/lib/qu/backend/base.rb +19 -0
- data/lib/qu/backend/spec.rb +265 -0
- data/lib/qu/failure.rb +4 -0
- data/lib/qu/job.rb +37 -0
- data/lib/qu/railtie.rb +9 -0
- data/lib/qu/tasks.rb +7 -0
- data/lib/qu/version.rb +3 -0
- data/lib/qu/worker.rb +63 -0
- data/qu.gemspec +22 -0
- data/spec/qu/job_spec.rb +78 -0
- data/spec/qu/worker_spec.rb +99 -0
- data/spec/qu_spec.rb +25 -0
- data/spec/spec_helper.rb +12 -0
- metadata +102 -0
data/.gitignore
ADDED
data/Gemfile
ADDED
@@ -0,0 +1,19 @@
|
|
1
|
+
source "http://rubygems.org"
|
2
|
+
gemspec :name => 'qu'
|
3
|
+
|
4
|
+
Dir['qu-*.gemspec'].each do |gemspec|
|
5
|
+
plugin = gemspec.scan(/qu-(.*)\.gemspec/).to_s
|
6
|
+
|
7
|
+
group plugin do
|
8
|
+
gemspec(:name => "qu-#{plugin}", :development_group => plugin)
|
9
|
+
end
|
10
|
+
end
|
11
|
+
|
12
|
+
group :test do
|
13
|
+
gem 'SystemTimer', :platform => :mri_18
|
14
|
+
gem 'ruby-debug', :platform => :mri_18
|
15
|
+
gem 'rake'
|
16
|
+
gem 'rspec', '~> 2.0'
|
17
|
+
gem 'guard-rspec'
|
18
|
+
gem 'guard-bundler'
|
19
|
+
end
|
data/Guardfile
ADDED
@@ -0,0 +1,12 @@
|
|
1
|
+
guard 'rspec', :version => 2 do
|
2
|
+
watch(%r{^spec/.+_spec\.rb$})
|
3
|
+
watch(%r{^lib/(.+)\.rb$}) { |m| "spec/#{m[1]}_spec.rb" }
|
4
|
+
watch(%r{^lib/qu/backend/spec\.rb$}) { |m| "spec/qu/backend" }
|
5
|
+
watch('spec/spec_helper.rb') { "spec" }
|
6
|
+
watch(%r{^spec/support/(.+)\.rb$}) { "spec" }
|
7
|
+
end
|
8
|
+
|
9
|
+
guard 'bundler' do
|
10
|
+
watch('Gemfile')
|
11
|
+
watch(/^.+\.gemspec/)
|
12
|
+
end
|
data/README.md
ADDED
@@ -0,0 +1,140 @@
|
|
1
|
+
# Qu
|
2
|
+
|
3
|
+
Qu is a Ruby library for queuing and processing background jobs. It is heavily inspired by delayed_job and Resque.
|
4
|
+
|
5
|
+
Qu was created to overcome some shortcomings in the existing queuing libraries that we experienced at [Ordered List](http://orderedlist.com) while building [SpeakerDeck](http://speakerdeck.com), [Gaug.es](http://get.gaug.es) and [Harmony](http://get.harmonyapp.com). The advantages of Qu are:
|
6
|
+
|
7
|
+
* Multiple backends (redis, mongo)
|
8
|
+
* Jobs are requeued when worker is killed
|
9
|
+
* Resque-like API
|
10
|
+
|
11
|
+
## Information & Help
|
12
|
+
|
13
|
+
* Find more information on the [Wiki](https://github.com/bkeepers/qu/wiki).
|
14
|
+
* Post to the [Google Group](http://groups.google.com/group/qu-users) for help or questions.
|
15
|
+
* See the [issue tracker](https://github.com/bkeepers/qu/issues) for known issues or to report an issue.
|
16
|
+
|
17
|
+
## Installation
|
18
|
+
|
19
|
+
### Rails
|
20
|
+
|
21
|
+
Decide which backend you want to use and add the gem for it to your `Gemfile`.
|
22
|
+
|
23
|
+
``` ruby
|
24
|
+
gem 'qu-redis'
|
25
|
+
```
|
26
|
+
|
27
|
+
That's all you need to do!
|
28
|
+
|
29
|
+
## Usage
|
30
|
+
|
31
|
+
Jobs are any class that responds to the `.perform` method:
|
32
|
+
|
33
|
+
``` ruby
|
34
|
+
class ProcessPresentation
|
35
|
+
def self.perform(presentation_id)
|
36
|
+
presentation = Presentation.find(presentation_id)
|
37
|
+
presentation.process!
|
38
|
+
end
|
39
|
+
end
|
40
|
+
```
|
41
|
+
|
42
|
+
You can add a job to the queue by calling the `enqueue` method:
|
43
|
+
|
44
|
+
``` ruby
|
45
|
+
job = Qu.enqueue ProcessPresentation, @presentation.id
|
46
|
+
puts "Enqueued job #{job.id}"
|
47
|
+
```
|
48
|
+
|
49
|
+
Any additional parameters passed to the `enqueue` method will be passed on to the `perform` method of your job class. These parameters will be stored in the backend, so they must be simple types that can easily be serialized and unserialized. So don't try to pass in an ActiveRecord object.
|
50
|
+
|
51
|
+
Processing the jobs on the queue can be done with a Rake task:
|
52
|
+
|
53
|
+
``` sh
|
54
|
+
$ bundle exec rake qu:work
|
55
|
+
```
|
56
|
+
|
57
|
+
You can easily inspect the queue or clear it:
|
58
|
+
|
59
|
+
``` ruby
|
60
|
+
puts "Jobs on the queue:", Qu.length
|
61
|
+
Qu.clear
|
62
|
+
```
|
63
|
+
|
64
|
+
### Queues
|
65
|
+
|
66
|
+
The `default` queue is used, um…by default. Jobs that don't specify a queue will be placed in that queue, and workers that don't specify a queue will work on that queue.
|
67
|
+
|
68
|
+
However, if you have some background jobs that are more or less important, or some that take longer than others, you may want to consider using multiple queues. You can have workers dedicated to specific queues, or simply tell all your workers to work on the most important queue first.
|
69
|
+
|
70
|
+
Jobs can be placed in a specific queue by setting the queue variable:
|
71
|
+
|
72
|
+
``` ruby
|
73
|
+
class CallThePresident
|
74
|
+
@queue = :urgent
|
75
|
+
|
76
|
+
def self.perform(message)
|
77
|
+
# …
|
78
|
+
end
|
79
|
+
end
|
80
|
+
```
|
81
|
+
|
82
|
+
You can then tell workers to work on this queue by passing an environment variable
|
83
|
+
|
84
|
+
``` sh
|
85
|
+
$ bundle exec rake qu:work QUEUES=urgent,default
|
86
|
+
```
|
87
|
+
|
88
|
+
Note that if you still want your worker to process the default queue, you must specify it. Queues will be process in the order they are specified.
|
89
|
+
|
90
|
+
You can also get the length or clear a specific queue:
|
91
|
+
|
92
|
+
``` ruby
|
93
|
+
Qu.length(:urgent)
|
94
|
+
Qu.clear(:urgent)
|
95
|
+
```
|
96
|
+
|
97
|
+
## Configuration
|
98
|
+
|
99
|
+
Most of the configuration for Qu should be automatic. It will also automatically detect ENV variables from Heroku for backend connections, so you shouldn't need to do anything to configure the backend.
|
100
|
+
|
101
|
+
However, if you do need to customize it, you can by calling the `Qu.configure`:
|
102
|
+
|
103
|
+
``` ruby
|
104
|
+
Qu.configure do |c|
|
105
|
+
c.connection = Redis::Namespace.new('myapp:qu', :redis => Redis.connect)
|
106
|
+
end
|
107
|
+
```
|
108
|
+
|
109
|
+
## Why another queuing library?
|
110
|
+
|
111
|
+
Resque and delayed_job are both great, but both of them have shortcomings that can be frustrating in production applications.
|
112
|
+
|
113
|
+
delayed_job was a brilliantly simple pioneer in the world of database-backed queues. While most asynchronous queuing systems were tending toward overly complex, it made use of your existing database and just worked. But there are a few flaws:
|
114
|
+
|
115
|
+
* Occasionally fails silently.
|
116
|
+
* Use of priority instead of separate named queues.
|
117
|
+
* Contention in the ActiveRecord backend with multiple workers. Occasionally the same job gets performed by multiple workers.
|
118
|
+
|
119
|
+
Resque, the wiser relative of delayed_job, fixes most of those issues. But in doing so, it forces some of its beliefs on you, and sometimes those beliefs just don't make sense for your environment. Here are some of the flaws of Resque:
|
120
|
+
|
121
|
+
* Redis is a great queue backend, but it doesn't make sense for every environment.
|
122
|
+
* Workers lose jobs when they are forced to quit. This has especially been an issue on Heroku.
|
123
|
+
* Forking before each job prevents memory leaks, but it is terribly inefficient in environments with a lot of fast jobs (the resque-jobs-per-fork plugin alleviates this)
|
124
|
+
|
125
|
+
Those shortcomings lead us to write Qu. It is not perfect, but we hope to overcome the issues we faced with other queuing libraries.
|
126
|
+
|
127
|
+
## Contributing
|
128
|
+
|
129
|
+
If you find what looks like a bug:
|
130
|
+
|
131
|
+
1. Search the [mailing list](http://groups.google.com/group/qu-users) to see if anyone else had the same issue.
|
132
|
+
2. Check the [GitHub issue tracker](http://github.com/bkeepers/qu/issues/) to see if anyone else has reported issue.
|
133
|
+
3. If you don't see anything, create an issue with information on how to reproduce it.
|
134
|
+
|
135
|
+
If you want to contribute an enhancement or a fix:
|
136
|
+
|
137
|
+
1. Fork the project on GitHub.
|
138
|
+
2. Make your changes with tests.
|
139
|
+
3. Commit the changes without making changes to the Rakefile, Gemfile, gemspec, or any other files that aren't related to your enhancement or fix
|
140
|
+
4. Send a pull request.
|
data/Rakefile
ADDED
@@ -0,0 +1,46 @@
|
|
1
|
+
# -*- encoding: utf-8 -*-
|
2
|
+
$:.push File.expand_path("../lib", __FILE__)
|
3
|
+
require "qu/version"
|
4
|
+
|
5
|
+
desc 'Build gem into the pkg directory'
|
6
|
+
task :build do
|
7
|
+
FileUtils.rm_rf('pkg')
|
8
|
+
Dir['*.gemspec'].each do |gemspec|
|
9
|
+
system "gem build #{gemspec}"
|
10
|
+
end
|
11
|
+
FileUtils.mkdir_p('pkg')
|
12
|
+
FileUtils.mv(Dir['*.gem'], 'pkg')
|
13
|
+
end
|
14
|
+
|
15
|
+
desc 'Tags version, pushes to remote, and pushes gem'
|
16
|
+
task :release => :build do
|
17
|
+
sh "git tag v#{Qu::VERSION}"
|
18
|
+
sh "git push origin master"
|
19
|
+
sh "git push origin v#{Qu::VERSION}"
|
20
|
+
sh "ls pkg/*.gem | xargs gem push"
|
21
|
+
end
|
22
|
+
|
23
|
+
require 'rspec/core/rake_task'
|
24
|
+
|
25
|
+
desc "Run all specs"
|
26
|
+
RSpec::Core::RakeTask.new(:spec) do |t|
|
27
|
+
t.rspec_opts = %w[--color]
|
28
|
+
t.verbose = false
|
29
|
+
end
|
30
|
+
|
31
|
+
namespace :spec do
|
32
|
+
Backends = %w(mongo redis)
|
33
|
+
|
34
|
+
Backends.each do |backend|
|
35
|
+
desc "Run specs for #{backend} backend"
|
36
|
+
RSpec::Core::RakeTask.new(backend) do |t|
|
37
|
+
t.rspec_opts = %w[--color]
|
38
|
+
t.verbose = false
|
39
|
+
t.pattern = "spec/qu/backend/#{backend}_spec.rb"
|
40
|
+
end
|
41
|
+
end
|
42
|
+
|
43
|
+
task :backends => Backends
|
44
|
+
end
|
45
|
+
|
46
|
+
task :default => :spec
|
data/lib/qu.rb
ADDED
@@ -0,0 +1,27 @@
|
|
1
|
+
require 'qu/version'
|
2
|
+
require 'qu/failure'
|
3
|
+
require 'qu/job'
|
4
|
+
require 'qu/backend/base'
|
5
|
+
|
6
|
+
require 'forwardable'
|
7
|
+
|
8
|
+
module Qu
|
9
|
+
autoload :Worker, 'qu/worker'
|
10
|
+
|
11
|
+
extend SingleForwardable
|
12
|
+
extend self
|
13
|
+
|
14
|
+
attr_accessor :backend, :failure
|
15
|
+
|
16
|
+
def_delegators :backend, :enqueue, :length, :queues, :reserve, :clear, :connection=
|
17
|
+
|
18
|
+
def backend
|
19
|
+
@backend || raise("Qu backend not configured. Install one of the backend gems like qu-redis.")
|
20
|
+
end
|
21
|
+
|
22
|
+
def configure(&block)
|
23
|
+
block.call(self)
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
require 'qu/railtie' if defined?(Rails)
|
@@ -0,0 +1,19 @@
|
|
1
|
+
require 'multi_json'
|
2
|
+
|
3
|
+
module Qu
|
4
|
+
module Backend
|
5
|
+
class Base
|
6
|
+
attr_accessor :connection
|
7
|
+
|
8
|
+
private
|
9
|
+
|
10
|
+
def encode(data)
|
11
|
+
MultiJson.encode(data) if data
|
12
|
+
end
|
13
|
+
|
14
|
+
def decode(data)
|
15
|
+
MultiJson.decode(data) if data
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
19
|
+
end
|
@@ -0,0 +1,265 @@
|
|
1
|
+
class SimpleJob
|
2
|
+
def self.perform
|
3
|
+
end
|
4
|
+
end
|
5
|
+
|
6
|
+
class CustomQueue
|
7
|
+
@queue = :custom
|
8
|
+
end
|
9
|
+
|
10
|
+
shared_examples_for 'a backend' do
|
11
|
+
let(:worker) { Qu::Worker.new('default') }
|
12
|
+
|
13
|
+
before(:all) do
|
14
|
+
Qu.backend = described_class.new
|
15
|
+
end
|
16
|
+
|
17
|
+
before do
|
18
|
+
subject.clear
|
19
|
+
subject.clear_workers
|
20
|
+
end
|
21
|
+
|
22
|
+
describe 'enqueue' do
|
23
|
+
it 'should return a job id' do
|
24
|
+
subject.enqueue(SimpleJob).should be_instance_of(Qu::Job)
|
25
|
+
end
|
26
|
+
|
27
|
+
it 'should add a job to the queue' do
|
28
|
+
job = subject.enqueue(SimpleJob)
|
29
|
+
job.queue.should == 'default'
|
30
|
+
subject.length(job.queue).should == 1
|
31
|
+
end
|
32
|
+
|
33
|
+
it 'should add queue to list of queues' do
|
34
|
+
subject.queues.should == []
|
35
|
+
job = subject.enqueue(SimpleJob)
|
36
|
+
subject.queues.should == [job.queue]
|
37
|
+
end
|
38
|
+
|
39
|
+
it 'should assign a different job id for the same job enqueue multiple times' do
|
40
|
+
subject.enqueue(SimpleJob).id.should_not == subject.enqueue(SimpleJob).id
|
41
|
+
end
|
42
|
+
end
|
43
|
+
|
44
|
+
describe 'length' do
|
45
|
+
it 'should use the default queue by default' do
|
46
|
+
subject.length.should == 0
|
47
|
+
subject.enqueue(SimpleJob)
|
48
|
+
subject.length.should == 1
|
49
|
+
end
|
50
|
+
end
|
51
|
+
|
52
|
+
describe 'clear' do
|
53
|
+
it 'should clear jobs for given queue' do
|
54
|
+
job = subject.enqueue SimpleJob
|
55
|
+
subject.length(job.queue).should == 1
|
56
|
+
subject.clear(job.queue)
|
57
|
+
subject.length(job.queue).should == 0
|
58
|
+
subject.queues.should_not include(job.queue)
|
59
|
+
end
|
60
|
+
|
61
|
+
it 'should not clear jobs for a different queue' do
|
62
|
+
job = subject.enqueue SimpleJob
|
63
|
+
subject.clear('other')
|
64
|
+
subject.length(job.queue).should == 1
|
65
|
+
end
|
66
|
+
|
67
|
+
it 'should clear all queues without any args' do
|
68
|
+
subject.enqueue(SimpleJob).queue.should == 'default'
|
69
|
+
subject.enqueue(CustomQueue).queue.should == 'custom'
|
70
|
+
subject.length('default').should == 1
|
71
|
+
subject.length('custom').should == 1
|
72
|
+
subject.clear
|
73
|
+
subject.length('default').should == 0
|
74
|
+
subject.length('custom').should == 0
|
75
|
+
end
|
76
|
+
|
77
|
+
it 'should clear failed queue without any args' do
|
78
|
+
job = subject.enqueue SimpleJob
|
79
|
+
subject.failed(job, Exception.new)
|
80
|
+
subject.length('failed').should == 1
|
81
|
+
subject.clear
|
82
|
+
subject.length('failed').should == 0
|
83
|
+
end
|
84
|
+
|
85
|
+
it 'should not clear failed queue with specified queues' do
|
86
|
+
job = subject.enqueue SimpleJob
|
87
|
+
subject.failed(job, Exception.new)
|
88
|
+
subject.length('failed').should == 1
|
89
|
+
subject.clear('default')
|
90
|
+
subject.length('failed').should == 1
|
91
|
+
end
|
92
|
+
end
|
93
|
+
|
94
|
+
describe 'reserve' do
|
95
|
+
before do
|
96
|
+
@job = subject.enqueue SimpleJob
|
97
|
+
end
|
98
|
+
|
99
|
+
it 'should return next job' do
|
100
|
+
subject.reserve(worker).id.should == @job.id
|
101
|
+
end
|
102
|
+
|
103
|
+
it 'should not return an already reserved job' do
|
104
|
+
another_job = subject.enqueue SimpleJob
|
105
|
+
subject.reserve(worker).id.should_not == subject.reserve(worker).id
|
106
|
+
end
|
107
|
+
|
108
|
+
it 'should return next job in given queues' do
|
109
|
+
subject.enqueue SimpleJob
|
110
|
+
job = subject.enqueue CustomQueue
|
111
|
+
subject.enqueue SimpleJob
|
112
|
+
|
113
|
+
worker = Qu::Worker.new('custom', 'default')
|
114
|
+
|
115
|
+
subject.reserve(worker).id.should == job.id
|
116
|
+
end
|
117
|
+
|
118
|
+
it 'should not return job from different queue' do
|
119
|
+
worker = Qu::Worker.new('video')
|
120
|
+
timeout { subject.reserve(worker) }.should be_nil
|
121
|
+
end
|
122
|
+
|
123
|
+
it 'should block by default if no jobs available' do
|
124
|
+
subject.clear
|
125
|
+
timeout(1) do
|
126
|
+
subject.reserve(worker)
|
127
|
+
fail("#reserve should block")
|
128
|
+
end
|
129
|
+
end
|
130
|
+
|
131
|
+
it 'should not block if :block option is set to false' do
|
132
|
+
timeout(1) do
|
133
|
+
subject.reserve(worker, :block => false)
|
134
|
+
true
|
135
|
+
end.should be_true
|
136
|
+
end
|
137
|
+
|
138
|
+
def timeout(count = 0.1, &block)
|
139
|
+
SystemTimer.timeout(count, &block)
|
140
|
+
rescue Timeout::Error
|
141
|
+
nil
|
142
|
+
end
|
143
|
+
end
|
144
|
+
|
145
|
+
describe 'failed' do
|
146
|
+
let(:job) { Qu::Job.new('1', SimpleJob, []) }
|
147
|
+
|
148
|
+
it 'should add to failure queue' do
|
149
|
+
subject.failed(job, Exception.new)
|
150
|
+
subject.length('failed').should == 1
|
151
|
+
end
|
152
|
+
|
153
|
+
it 'should not add failed queue to the list of queues' do
|
154
|
+
subject.failed(job, Exception.new)
|
155
|
+
subject.queues.should_not include('failed')
|
156
|
+
end
|
157
|
+
end
|
158
|
+
|
159
|
+
describe 'completed' do
|
160
|
+
it 'should be defined' do
|
161
|
+
subject.respond_to?(:completed).should be_true
|
162
|
+
end
|
163
|
+
end
|
164
|
+
|
165
|
+
describe 'release' do
|
166
|
+
before do
|
167
|
+
subject.enqueue SimpleJob
|
168
|
+
end
|
169
|
+
|
170
|
+
it 'should add the job back on the queue' do
|
171
|
+
job = subject.reserve(worker)
|
172
|
+
subject.length(job.queue).should == 0
|
173
|
+
subject.release(job)
|
174
|
+
subject.length(job.queue).should == 1
|
175
|
+
end
|
176
|
+
end
|
177
|
+
|
178
|
+
describe 'requeue' do
|
179
|
+
context 'with a failed job' do
|
180
|
+
before do
|
181
|
+
subject.enqueue(SimpleJob)
|
182
|
+
@job = subject.reserve(worker)
|
183
|
+
subject.failed(@job, Exception.new)
|
184
|
+
end
|
185
|
+
|
186
|
+
it 'should add the job back on the queue' do
|
187
|
+
subject.length(@job.queue).should == 0
|
188
|
+
subject.requeue(@job.id)
|
189
|
+
subject.length(@job.queue).should == 1
|
190
|
+
|
191
|
+
job = subject.reserve(worker)
|
192
|
+
job.should be_instance_of(Qu::Job)
|
193
|
+
job.id.should == @job.id
|
194
|
+
job.klass.should == @job.klass
|
195
|
+
job.args.should == @job.args
|
196
|
+
end
|
197
|
+
|
198
|
+
it 'should remove the job from the failed jobs' do
|
199
|
+
subject.length('failed').should == 1
|
200
|
+
subject.requeue(@job.id)
|
201
|
+
subject.length('failed').should == 0
|
202
|
+
end
|
203
|
+
|
204
|
+
it 'should return the job' do
|
205
|
+
subject.requeue(@job.id).id.should == @job.id
|
206
|
+
end
|
207
|
+
end
|
208
|
+
|
209
|
+
context 'without a failed job' do
|
210
|
+
it 'should return false' do
|
211
|
+
subject.requeue('1').should be_false
|
212
|
+
end
|
213
|
+
end
|
214
|
+
end
|
215
|
+
|
216
|
+
describe 'register_worker' do
|
217
|
+
let(:worker) { Qu::Worker.new('default') }
|
218
|
+
|
219
|
+
it 'should add worker to array of workers' do
|
220
|
+
subject.register_worker(worker)
|
221
|
+
subject.workers.size.should == 1
|
222
|
+
subject.workers.first.attributes.should == worker.attributes
|
223
|
+
end
|
224
|
+
end
|
225
|
+
|
226
|
+
describe 'clear_workers' do
|
227
|
+
before { subject.register_worker Qu::Worker.new('default') }
|
228
|
+
|
229
|
+
it 'should remove workers' do
|
230
|
+
subject.workers.size.should == 1
|
231
|
+
subject.clear_workers
|
232
|
+
subject.workers.size.should == 0
|
233
|
+
end
|
234
|
+
end
|
235
|
+
|
236
|
+
describe 'unregister_worker' do
|
237
|
+
before { subject.register_worker Qu::Worker.new('default') }
|
238
|
+
|
239
|
+
it 'should remove worker' do
|
240
|
+
subject.unregister_worker(worker.id)
|
241
|
+
subject.workers.size.should == 0
|
242
|
+
end
|
243
|
+
|
244
|
+
it 'should not remove other workers' do
|
245
|
+
other_worker = Qu::Worker.new('other')
|
246
|
+
subject.register_worker(other_worker)
|
247
|
+
subject.workers.size.should == 2
|
248
|
+
subject.unregister_worker(other_worker.id)
|
249
|
+
subject.workers.size.should == 1
|
250
|
+
subject.workers.first.id.should == worker.id
|
251
|
+
end
|
252
|
+
end
|
253
|
+
|
254
|
+
describe 'connection=' do
|
255
|
+
it 'should allow setting the connection' do
|
256
|
+
connection = mock('a connection')
|
257
|
+
subject.connection = connection
|
258
|
+
subject.connection.should == connection
|
259
|
+
end
|
260
|
+
|
261
|
+
it 'should provide a default connection' do
|
262
|
+
subject.connection.should_not be_nil
|
263
|
+
end
|
264
|
+
end
|
265
|
+
end
|
data/lib/qu/failure.rb
ADDED
data/lib/qu/job.rb
ADDED
@@ -0,0 +1,37 @@
|
|
1
|
+
module Qu
|
2
|
+
class Job
|
3
|
+
attr_accessor :id, :klass, :args
|
4
|
+
|
5
|
+
def initialize(id, klass, args)
|
6
|
+
@id, @args = id, args
|
7
|
+
|
8
|
+
@klass = klass.is_a?(Class) ? klass : constantize(klass)
|
9
|
+
end
|
10
|
+
|
11
|
+
def queue
|
12
|
+
(klass.instance_variable_get(:@queue) || 'default').to_s
|
13
|
+
end
|
14
|
+
|
15
|
+
def perform
|
16
|
+
klass.perform(*args)
|
17
|
+
Qu.backend.completed(self)
|
18
|
+
rescue Qu::Worker::Abort
|
19
|
+
Qu.backend.release(self)
|
20
|
+
raise
|
21
|
+
rescue Exception => e
|
22
|
+
Qu.failure.create(self, e) if Qu.failure
|
23
|
+
Qu.backend.failed(self, e)
|
24
|
+
end
|
25
|
+
|
26
|
+
protected
|
27
|
+
|
28
|
+
def constantize(class_name)
|
29
|
+
constant = Object
|
30
|
+
class_name.split('::').each do |name|
|
31
|
+
constant = constant.const_get(name) || constant.const_missing(name)
|
32
|
+
end
|
33
|
+
constant
|
34
|
+
end
|
35
|
+
|
36
|
+
end
|
37
|
+
end
|
data/lib/qu/railtie.rb
ADDED
data/lib/qu/tasks.rb
ADDED
data/lib/qu/version.rb
ADDED
data/lib/qu/worker.rb
ADDED
@@ -0,0 +1,63 @@
|
|
1
|
+
module Qu
|
2
|
+
class Worker
|
3
|
+
attr_accessor :queues
|
4
|
+
|
5
|
+
class Abort < Exception
|
6
|
+
end
|
7
|
+
|
8
|
+
def initialize(*queues)
|
9
|
+
@queues = queues.flatten
|
10
|
+
self.attributes = @queues.pop if @queues.last.is_a?(Hash)
|
11
|
+
@queues << 'default' if @queues.empty?
|
12
|
+
end
|
13
|
+
|
14
|
+
def attributes=(attrs)
|
15
|
+
attrs.each do |attr, value|
|
16
|
+
self.instance_variable_set("@#{attr}", value)
|
17
|
+
end
|
18
|
+
end
|
19
|
+
|
20
|
+
def attributes
|
21
|
+
{'hostname' => hostname, 'pid' => pid, 'queues' => queues}
|
22
|
+
end
|
23
|
+
|
24
|
+
def handle_signals
|
25
|
+
%W(INT TERM).each do |sig|
|
26
|
+
trap(sig) { raise Abort }
|
27
|
+
end
|
28
|
+
end
|
29
|
+
|
30
|
+
def work_off
|
31
|
+
while job = Qu.reserve(self, :block => false)
|
32
|
+
job.perform
|
33
|
+
end
|
34
|
+
end
|
35
|
+
|
36
|
+
def work
|
37
|
+
job = Qu.reserve(self)
|
38
|
+
job.perform
|
39
|
+
end
|
40
|
+
|
41
|
+
def start
|
42
|
+
handle_signals
|
43
|
+
Qu.backend.register_worker(self)
|
44
|
+
loop { work }
|
45
|
+
rescue Abort => e
|
46
|
+
# Ok, we'll shut down, but give us a sec
|
47
|
+
ensure
|
48
|
+
Qu.backend.unregister_worker(self)
|
49
|
+
end
|
50
|
+
|
51
|
+
def id
|
52
|
+
"#{hostname}:#{pid}:#{queues.join(',')}"
|
53
|
+
end
|
54
|
+
|
55
|
+
def pid
|
56
|
+
@pid ||= Process.pid
|
57
|
+
end
|
58
|
+
|
59
|
+
def hostname
|
60
|
+
@hostname ||= `hostname`.strip
|
61
|
+
end
|
62
|
+
end
|
63
|
+
end
|
data/qu.gemspec
ADDED
@@ -0,0 +1,22 @@
|
|
1
|
+
# -*- encoding: utf-8 -*-
|
2
|
+
$:.push File.expand_path("../lib", __FILE__)
|
3
|
+
require "qu/version"
|
4
|
+
|
5
|
+
plugins = Dir['qu-*.gemspec'].map {|gemspec| gemspec.scan(/qu-(.*)\.gemspec/).to_s }.join('\|')
|
6
|
+
|
7
|
+
Gem::Specification.new do |s|
|
8
|
+
s.name = "qu"
|
9
|
+
s.version = Qu::VERSION
|
10
|
+
s.authors = ["Brandon Keepers"]
|
11
|
+
s.email = ["brandon@opensoul.org"]
|
12
|
+
s.homepage = "http://github.com/bkeepers/qu"
|
13
|
+
s.summary = %q{}
|
14
|
+
s.description = %q{}
|
15
|
+
|
16
|
+
s.files = `git ls-files | grep -v '#{plugins}'`.split("\n")
|
17
|
+
s.test_files = `git ls-files -- spec | grep -v '#{plugins}'`.split("\n")
|
18
|
+
s.executables = `git ls-files -- bin`.split("\n").map{ |f| File.basename(f) }
|
19
|
+
s.require_paths = ["lib"]
|
20
|
+
|
21
|
+
s.add_dependency 'multi_json'
|
22
|
+
end
|
data/spec/qu/job_spec.rb
ADDED
@@ -0,0 +1,78 @@
|
|
1
|
+
require 'spec_helper'
|
2
|
+
|
3
|
+
describe Qu::Job do
|
4
|
+
class MyJob
|
5
|
+
@queue = :custom
|
6
|
+
end
|
7
|
+
|
8
|
+
describe 'queue' do
|
9
|
+
it 'should default to "default"' do
|
10
|
+
Qu::Job.new('1', SimpleJob, []).queue.should == 'default'
|
11
|
+
end
|
12
|
+
|
13
|
+
it 'should get queue from job instance variable' do
|
14
|
+
Qu::Job.new('1', MyJob, []).queue.should == 'custom'
|
15
|
+
end
|
16
|
+
end
|
17
|
+
|
18
|
+
describe 'klass' do
|
19
|
+
it 'should constantize string' do
|
20
|
+
Qu::Job.new('1', 'MyJob', []).klass.should == MyJob
|
21
|
+
end
|
22
|
+
|
23
|
+
it 'should find namespaced jobs' do
|
24
|
+
Qu::Job.new('1', 'Qu::Job', []).klass.should == Qu::Job
|
25
|
+
end
|
26
|
+
end
|
27
|
+
|
28
|
+
describe 'perform' do
|
29
|
+
subject { Qu::Job.new('1', SimpleJob, []) }
|
30
|
+
|
31
|
+
it 'should call .perform on job class with args' do
|
32
|
+
subject.args = ['a', 'b']
|
33
|
+
SimpleJob.should_receive(:perform).with('a', 'b')
|
34
|
+
subject.perform
|
35
|
+
end
|
36
|
+
|
37
|
+
it 'should call completed on backend' do
|
38
|
+
Qu.backend.should_receive(:completed)
|
39
|
+
subject.perform
|
40
|
+
end
|
41
|
+
|
42
|
+
context 'when being aborted' do
|
43
|
+
before do
|
44
|
+
SimpleJob.stub(:perform).and_raise(Qu::Worker::Abort)
|
45
|
+
end
|
46
|
+
|
47
|
+
it 'should release the job and re-raise the error' do
|
48
|
+
Qu.backend.should_receive(:release).with(subject)
|
49
|
+
lambda { subject.perform }.should raise_error(Qu::Worker::Abort)
|
50
|
+
end
|
51
|
+
end
|
52
|
+
|
53
|
+
context 'when the job raises an error' do
|
54
|
+
let(:error) { Exception.new("Some kind of error") }
|
55
|
+
|
56
|
+
before do
|
57
|
+
SimpleJob.stub!(:perform).and_raise(error)
|
58
|
+
end
|
59
|
+
|
60
|
+
it 'should call failed on backend' do
|
61
|
+
Qu.backend.should_receive(:failed).with(subject, error)
|
62
|
+
subject.perform
|
63
|
+
end
|
64
|
+
|
65
|
+
it 'should not call completed on backend' do
|
66
|
+
Qu.backend.should_not_receive(:completed)
|
67
|
+
subject.perform
|
68
|
+
end
|
69
|
+
|
70
|
+
it 'should call create on failure backend' do
|
71
|
+
Qu.failure = mock('a failure backend')
|
72
|
+
Qu.failure.should_receive(:create).with(subject, error)
|
73
|
+
subject.perform
|
74
|
+
end
|
75
|
+
end
|
76
|
+
|
77
|
+
end
|
78
|
+
end
|
@@ -0,0 +1,99 @@
|
|
1
|
+
require 'spec_helper'
|
2
|
+
|
3
|
+
describe Qu::Worker do
|
4
|
+
let(:job) { Qu::Job.new('1', SimpleJob, []) }
|
5
|
+
|
6
|
+
describe 'queues' do
|
7
|
+
it 'should use default if none specified' do
|
8
|
+
Qu::Worker.new.queues.should == ['default']
|
9
|
+
Qu::Worker.new('default').queues.should == ['default']
|
10
|
+
Qu::Worker.new(['default']).queues.should == ['default']
|
11
|
+
end
|
12
|
+
end
|
13
|
+
|
14
|
+
describe 'work' do
|
15
|
+
before do
|
16
|
+
Qu.stub!(:reserve).and_return(job)
|
17
|
+
end
|
18
|
+
|
19
|
+
it 'should reserve a job' do
|
20
|
+
Qu.should_receive(:reserve).with(subject).and_return(job)
|
21
|
+
subject.work
|
22
|
+
end
|
23
|
+
|
24
|
+
it 'should perform the job' do
|
25
|
+
job.should_receive(:perform)
|
26
|
+
subject.work
|
27
|
+
end
|
28
|
+
end
|
29
|
+
|
30
|
+
describe 'work_off' do
|
31
|
+
it 'should work all jobs off the queue' do
|
32
|
+
Qu.should_receive(:reserve).exactly(4).times.with(subject, :block => false).and_return(job, job, job, nil)
|
33
|
+
subject.work_off
|
34
|
+
end
|
35
|
+
end
|
36
|
+
|
37
|
+
describe 'start' do
|
38
|
+
before do
|
39
|
+
subject.stub(:loop)
|
40
|
+
end
|
41
|
+
|
42
|
+
it 'should register worker' do
|
43
|
+
Qu.backend.should_receive(:register_worker).with(subject)
|
44
|
+
subject.start
|
45
|
+
end
|
46
|
+
|
47
|
+
context 'when aborting' do
|
48
|
+
before do
|
49
|
+
subject.stub(:loop).and_raise(Qu::Worker::Abort)
|
50
|
+
end
|
51
|
+
|
52
|
+
it 'should unregister worker' do
|
53
|
+
Qu.backend.should_receive(:unregister_worker).with(subject)
|
54
|
+
subject.start
|
55
|
+
end
|
56
|
+
end
|
57
|
+
end
|
58
|
+
|
59
|
+
describe 'pid' do
|
60
|
+
it 'should equal process id' do
|
61
|
+
subject.pid.should == Process.pid
|
62
|
+
end
|
63
|
+
|
64
|
+
it 'should use provided pid' do
|
65
|
+
Qu::Worker.new(:pid => 1).pid.should == 1
|
66
|
+
end
|
67
|
+
end
|
68
|
+
|
69
|
+
describe 'id' do
|
70
|
+
it 'should return hostname, pid, and queues' do
|
71
|
+
worker = Qu::Worker.new('a', 'b', :hostname => 'quspec', :pid => 123)
|
72
|
+
worker.id.should == 'quspec:123:a,b'
|
73
|
+
end
|
74
|
+
|
75
|
+
it 'should not expand star in queue names' do
|
76
|
+
Qu::Worker.new('a', '*').id.should =~ /a,*/
|
77
|
+
end
|
78
|
+
end
|
79
|
+
|
80
|
+
describe 'hostname' do
|
81
|
+
it 'should get hostname' do
|
82
|
+
subject.hostname.should_not be_empty
|
83
|
+
end
|
84
|
+
|
85
|
+
it 'should use provided hostname' do
|
86
|
+
Qu::Worker.new(:hostname => 'quspec').hostname.should == 'quspec'
|
87
|
+
end
|
88
|
+
end
|
89
|
+
|
90
|
+
describe 'attributes' do
|
91
|
+
let(:attrs) do
|
92
|
+
{'hostname' => 'omgbbq', 'pid' => 987, 'queues' => ['a', '*']}
|
93
|
+
end
|
94
|
+
|
95
|
+
it 'should return hash of attributes' do
|
96
|
+
Qu::Worker.new(attrs).attributes.should == attrs
|
97
|
+
end
|
98
|
+
end
|
99
|
+
end
|
data/spec/qu_spec.rb
ADDED
@@ -0,0 +1,25 @@
|
|
1
|
+
require 'spec_helper'
|
2
|
+
|
3
|
+
describe Qu do
|
4
|
+
%w(enqueue length queues reserve clear connection=).each do |method|
|
5
|
+
it "should delegate #{method} to backend" do
|
6
|
+
Qu.backend.should_receive(method).with(:arg)
|
7
|
+
Qu.send(method, :arg)
|
8
|
+
end
|
9
|
+
end
|
10
|
+
|
11
|
+
describe 'configure' do
|
12
|
+
it 'should yield Qu' do
|
13
|
+
Qu.configure do |c|
|
14
|
+
c.should == Qu
|
15
|
+
end
|
16
|
+
end
|
17
|
+
end
|
18
|
+
|
19
|
+
describe 'backend' do
|
20
|
+
it 'should raise error if backend not configured' do
|
21
|
+
Qu.backend = nil
|
22
|
+
lambda { Qu.backend }.should raise_error
|
23
|
+
end
|
24
|
+
end
|
25
|
+
end
|
data/spec/spec_helper.rb
ADDED
@@ -0,0 +1,12 @@
|
|
1
|
+
require 'bundler'
|
2
|
+
Bundler.require :default, :test
|
3
|
+
require 'qu'
|
4
|
+
require 'qu/backend/spec'
|
5
|
+
|
6
|
+
RSpec.configure do |config|
|
7
|
+
config.before do
|
8
|
+
Qu.backend = mock('a backend', :reserve => nil, :failed => nil, :completed => nil,
|
9
|
+
:register_worker => nil, :unregister_worker => nil)
|
10
|
+
Qu.failure = nil
|
11
|
+
end
|
12
|
+
end
|
metadata
ADDED
@@ -0,0 +1,102 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: qu
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
hash: 27
|
5
|
+
prerelease:
|
6
|
+
segments:
|
7
|
+
- 0
|
8
|
+
- 1
|
9
|
+
- 0
|
10
|
+
version: 0.1.0
|
11
|
+
platform: ruby
|
12
|
+
authors:
|
13
|
+
- Brandon Keepers
|
14
|
+
autorequire:
|
15
|
+
bindir: bin
|
16
|
+
cert_chain: []
|
17
|
+
|
18
|
+
date: 2011-09-23 00:00:00 -04:00
|
19
|
+
default_executable:
|
20
|
+
dependencies:
|
21
|
+
- !ruby/object:Gem::Dependency
|
22
|
+
name: multi_json
|
23
|
+
prerelease: false
|
24
|
+
requirement: &id001 !ruby/object:Gem::Requirement
|
25
|
+
none: false
|
26
|
+
requirements:
|
27
|
+
- - ">="
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
hash: 3
|
30
|
+
segments:
|
31
|
+
- 0
|
32
|
+
version: "0"
|
33
|
+
type: :runtime
|
34
|
+
version_requirements: *id001
|
35
|
+
description: ""
|
36
|
+
email:
|
37
|
+
- brandon@opensoul.org
|
38
|
+
executables: []
|
39
|
+
|
40
|
+
extensions: []
|
41
|
+
|
42
|
+
extra_rdoc_files: []
|
43
|
+
|
44
|
+
files:
|
45
|
+
- .gitignore
|
46
|
+
- Gemfile
|
47
|
+
- Guardfile
|
48
|
+
- README.md
|
49
|
+
- Rakefile
|
50
|
+
- lib/qu.rb
|
51
|
+
- lib/qu/backend/base.rb
|
52
|
+
- lib/qu/backend/spec.rb
|
53
|
+
- lib/qu/failure.rb
|
54
|
+
- lib/qu/job.rb
|
55
|
+
- lib/qu/railtie.rb
|
56
|
+
- lib/qu/tasks.rb
|
57
|
+
- lib/qu/version.rb
|
58
|
+
- lib/qu/worker.rb
|
59
|
+
- qu.gemspec
|
60
|
+
- spec/qu/job_spec.rb
|
61
|
+
- spec/qu/worker_spec.rb
|
62
|
+
- spec/qu_spec.rb
|
63
|
+
- spec/spec_helper.rb
|
64
|
+
has_rdoc: true
|
65
|
+
homepage: http://github.com/bkeepers/qu
|
66
|
+
licenses: []
|
67
|
+
|
68
|
+
post_install_message:
|
69
|
+
rdoc_options: []
|
70
|
+
|
71
|
+
require_paths:
|
72
|
+
- lib
|
73
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
74
|
+
none: false
|
75
|
+
requirements:
|
76
|
+
- - ">="
|
77
|
+
- !ruby/object:Gem::Version
|
78
|
+
hash: 3
|
79
|
+
segments:
|
80
|
+
- 0
|
81
|
+
version: "0"
|
82
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
83
|
+
none: false
|
84
|
+
requirements:
|
85
|
+
- - ">="
|
86
|
+
- !ruby/object:Gem::Version
|
87
|
+
hash: 3
|
88
|
+
segments:
|
89
|
+
- 0
|
90
|
+
version: "0"
|
91
|
+
requirements: []
|
92
|
+
|
93
|
+
rubyforge_project:
|
94
|
+
rubygems_version: 1.6.1
|
95
|
+
signing_key:
|
96
|
+
specification_version: 3
|
97
|
+
summary: ""
|
98
|
+
test_files:
|
99
|
+
- spec/qu/job_spec.rb
|
100
|
+
- spec/qu/worker_spec.rb
|
101
|
+
- spec/qu_spec.rb
|
102
|
+
- spec/spec_helper.rb
|