nfo-resque-mongo 1.15.1 → 1.17.1
Sign up to get free protection for your applications and to get access to all the features.
- data/HISTORY.md +32 -0
- data/README.markdown +135 -62
- data/Rakefile +6 -2
- data/lib/nfo-resque-mongo.rb +2 -0
- data/lib/resque.rb +20 -6
- data/lib/resque/failure/hoptoad.rb +22 -113
- data/lib/resque/failure/mongo.rb +6 -1
- data/lib/resque/failure/multiple.rb +5 -1
- data/lib/resque/failure/redis.rb +51 -0
- data/lib/resque/helpers.rb +13 -17
- data/lib/resque/plugin.rb +5 -0
- data/lib/resque/server.rb +18 -8
- data/lib/resque/server/public/favicon.ico +0 -0
- data/lib/resque/server/public/style.css +2 -2
- data/lib/resque/server/views/failed.erb +3 -2
- data/lib/resque/server/views/queues.erb +1 -1
- data/lib/resque/server/views/stats.erb +15 -1
- data/lib/resque/server/views/working.erb +10 -6
- data/lib/resque/version.rb +1 -1
- data/lib/resque/worker.rb +9 -4
- data/{tasks → lib/tasks}/redis.rake +0 -0
- data/{tasks → lib/tasks}/resque.rake +0 -0
- data/test/hoptoad_test.rb +25 -0
- data/test/job_hooks_test.rb +40 -0
- data/test/queue_stats_test.rb +1 -1
- data/test/resque_test.rb +1 -1
- data/test/test_helper.rb +5 -2
- metadata +66 -61
data/HISTORY.md
CHANGED
@@ -1,3 +1,35 @@
|
|
1
|
+
## 1.17.1 (2011-05-27)
|
2
|
+
|
3
|
+
* Reverted `exit` change. Back to `exit!`.
|
4
|
+
|
5
|
+
## 1.17.0 (2011-05-26)
|
6
|
+
|
7
|
+
* Workers exit with `exit` instead of `exit!`. This means you
|
8
|
+
can now use `at_exit` hooks inside workers.
|
9
|
+
* More monit typo fixes.
|
10
|
+
* Fixed bug in Hoptoad backend.
|
11
|
+
* Web UI: Wrap preformatted arguments.
|
12
|
+
|
13
|
+
## 1.16.1 (2011-05-17)
|
14
|
+
|
15
|
+
* Bugfix: Resque::Failure::Hoptoad.configure works again
|
16
|
+
* Bugfix: Loading rake tasks
|
17
|
+
|
18
|
+
## 1.16.0 (2011-05-16)
|
19
|
+
|
20
|
+
* Optional Hoptoad backend extracted into hoptoad_notifier. Install the gem to use it.
|
21
|
+
* Added `Worker#paused?` method
|
22
|
+
* Bugfix: Properly reseed random number generator after forking.
|
23
|
+
* Bugfix: Resque.redis=(<a Redis::Namespace>)
|
24
|
+
* Bugfix: Monit example stdout/stderr redirection
|
25
|
+
* Bugfix: Removing single failure now works with multiple failure backends
|
26
|
+
* Web: 'Remove Queue' now requires confirmation
|
27
|
+
* Web: Favicon!
|
28
|
+
* Web Bugfix: Dates display in Safari
|
29
|
+
* Web Bugfix: Dates display timezone
|
30
|
+
* Web Bugfix: Race condition querying working workers
|
31
|
+
* Web Bugfix: Fix polling /workers/all in resque-web
|
32
|
+
|
1
33
|
## 1.15.0 (2011-03-18)
|
2
34
|
|
3
35
|
* Fallback to Redis.connect. Makes ENV variables and whatnot work.
|
data/README.markdown
CHANGED
@@ -6,8 +6,7 @@ library functionality has been ported (all tests pass) and the
|
|
6
6
|
monitoring sinatra app works except the "stats" panel, although there are
|
7
7
|
a lot of details that need to been taken care of.
|
8
8
|
|
9
|
-
Resque-mongo uses a fairly
|
10
|
-
findAndModify that is only supported by 0.20+ version ofthe ruby mongo driver.
|
9
|
+
Resque-mongo uses a fairly nice feature of mongo, [findAndModify()][fnr] that is only supported by 0.20+ version of the Ruby mongo driver.
|
11
10
|
|
12
11
|
Also, check your mongo version: 1.3.0 or higher
|
13
12
|
|
@@ -17,6 +16,7 @@ Also, check your mongo version: 1.3.0 or higher
|
|
17
16
|
# Rails integration, config/environment.rb
|
18
17
|
config.gem 'nfo-resque-mongo', :lib => 'resque'
|
19
18
|
|
19
|
+
To make the regular merge with the [official Resque repository](https://github.com/defunkt/resque/) easier, some redis-related files were kept.
|
20
20
|
|
21
21
|
Resque
|
22
22
|
======
|
@@ -67,14 +67,17 @@ later, pull those jobs off the queue and process them.
|
|
67
67
|
Resque jobs are Ruby classes (or modules) which respond to the
|
68
68
|
`perform` method. Here's an example:
|
69
69
|
|
70
|
-
class Archive
|
71
|
-
@queue = :file_serve
|
72
70
|
|
73
|
-
|
74
|
-
|
75
|
-
|
76
|
-
|
77
|
-
|
71
|
+
``` ruby
|
72
|
+
class Archive
|
73
|
+
@queue = :file_serve
|
74
|
+
|
75
|
+
def self.perform(repo_id, branch = 'master')
|
76
|
+
repo = Repository.find(repo_id)
|
77
|
+
repo.create_archive(branch)
|
78
|
+
end
|
79
|
+
end
|
80
|
+
```
|
78
81
|
|
79
82
|
The `@queue` class instance variable determines which queue `Archive`
|
80
83
|
jobs will be placed in. Queues are arbitrary and created on the fly -
|
@@ -83,11 +86,13 @@ you can name them whatever you want and have as many as you want.
|
|
83
86
|
To place an `Archive` job on the `file_serve` queue, we might add this
|
84
87
|
to our application's pre-existing `Repository` class:
|
85
88
|
|
86
|
-
|
87
|
-
|
88
|
-
|
89
|
-
|
90
|
-
|
89
|
+
``` ruby
|
90
|
+
class Repository
|
91
|
+
def async_create_archive(branch)
|
92
|
+
Resque.enqueue(Archive, self.id, branch)
|
93
|
+
end
|
94
|
+
end
|
95
|
+
```
|
91
96
|
|
92
97
|
Now when we call `repo.async_create_archive('masterbrew')` in our
|
93
98
|
application, a job will be created and placed on the `file_serve`
|
@@ -95,12 +100,16 @@ queue.
|
|
95
100
|
|
96
101
|
Later, a worker will run something like this code to process the job:
|
97
102
|
|
98
|
-
|
99
|
-
|
103
|
+
``` ruby
|
104
|
+
klass, args = Resque.reserve(:file_serve)
|
105
|
+
klass.perform(*args) if klass.respond_to? :perform
|
106
|
+
```
|
100
107
|
|
101
108
|
Which translates to:
|
102
109
|
|
103
|
-
|
110
|
+
``` ruby
|
111
|
+
Archive.perform(44, 'masterbrew')
|
112
|
+
```
|
104
113
|
|
105
114
|
Let's start a worker to run `file_serve` jobs:
|
106
115
|
|
@@ -149,25 +158,33 @@ needs to be crunched later into a queue.
|
|
149
158
|
Jobs are persisted to queues as JSON objects. Let's take our `Archive`
|
150
159
|
example from above. We'll run the following code to create a job:
|
151
160
|
|
152
|
-
|
153
|
-
|
161
|
+
``` ruby
|
162
|
+
repo = Repository.find(44)
|
163
|
+
repo.async_create_archive('masterbrew')
|
164
|
+
```
|
154
165
|
|
155
166
|
The following JSON will be stored in the `file_serve` queue:
|
156
167
|
|
157
|
-
|
158
|
-
|
159
|
-
|
160
|
-
|
168
|
+
``` javascript
|
169
|
+
{
|
170
|
+
'class': 'Archive',
|
171
|
+
'args': [ 44, 'masterbrew' ]
|
172
|
+
}
|
173
|
+
```
|
161
174
|
|
162
175
|
Because of this your jobs must only accept arguments that can be JSON encoded.
|
163
176
|
|
164
177
|
So instead of doing this:
|
165
178
|
|
166
|
-
|
179
|
+
``` ruby
|
180
|
+
Resque.enqueue(Archive, self, branch)
|
181
|
+
```
|
167
182
|
|
168
183
|
do this:
|
169
184
|
|
170
|
-
|
185
|
+
``` ruby
|
186
|
+
Resque.enqueue(Archive, self.id, branch)
|
187
|
+
```
|
171
188
|
|
172
189
|
This is why our above example (and all the examples in `examples/`)
|
173
190
|
uses object IDs instead of passing around the objects.
|
@@ -207,15 +224,17 @@ Workers
|
|
207
224
|
|
208
225
|
Resque workers are rake tasks that run forever. They basically do this:
|
209
226
|
|
210
|
-
|
211
|
-
|
212
|
-
|
213
|
-
|
214
|
-
|
215
|
-
|
216
|
-
|
217
|
-
|
218
|
-
|
227
|
+
``` ruby
|
228
|
+
start
|
229
|
+
loop do
|
230
|
+
if job = reserve
|
231
|
+
job.process
|
232
|
+
else
|
233
|
+
sleep 5
|
234
|
+
end
|
235
|
+
end
|
236
|
+
shutdown
|
237
|
+
```
|
219
238
|
|
220
239
|
Starting a worker is simple. Here's our example from earlier:
|
221
240
|
|
@@ -234,13 +253,17 @@ This will load the environment before starting a worker. Alternately
|
|
234
253
|
we can define a `resque:setup` task with a dependency on the
|
235
254
|
`environment` rake task:
|
236
255
|
|
237
|
-
|
256
|
+
``` ruby
|
257
|
+
task "resque:setup" => :environment
|
258
|
+
```
|
238
259
|
|
239
260
|
GitHub's setup task looks like this:
|
240
261
|
|
241
|
-
|
242
|
-
|
243
|
-
|
262
|
+
``` ruby
|
263
|
+
task "resque:setup" => :environment do
|
264
|
+
Grit::Git.git_timeout = 10.minutes
|
265
|
+
end
|
266
|
+
```
|
244
267
|
|
245
268
|
We don't want the `git_timeout` as high as 10 minutes in our web app,
|
246
269
|
but in the Resque workers it's fine.
|
@@ -421,7 +444,7 @@ The Front End
|
|
421
444
|
Resque comes with a Sinatra-based front end for seeing what's up with
|
422
445
|
your queue.
|
423
446
|
|
424
|
-
![The Front End](
|
447
|
+
![The Front End](https://img.skitch.com/20110528-pc67a8qsfapgjxf5gagxd92fcu.png)
|
425
448
|
|
426
449
|
### Standalone
|
427
450
|
|
@@ -455,11 +478,13 @@ Nginx: <http://www.modrails.com/documentation/Users%20guide%20Nginx.html#deployi
|
|
455
478
|
If you want to load Resque on a subpath, possibly alongside other
|
456
479
|
apps, it's easy to do with Rack's `URLMap`:
|
457
480
|
|
458
|
-
|
481
|
+
``` ruby
|
482
|
+
require 'resque/server'
|
459
483
|
|
460
|
-
|
461
|
-
|
462
|
-
|
484
|
+
run Rack::URLMap.new \
|
485
|
+
"/" => Your::App.new,
|
486
|
+
"/resque" => Resque::Server.new
|
487
|
+
```
|
463
488
|
|
464
489
|
Check `examples/demo/config.ru` for a functional example (including
|
465
490
|
HTTP basic auth).
|
@@ -555,13 +580,8 @@ together. But, it's not that hard.
|
|
555
580
|
Resque Dependencies
|
556
581
|
-------------------
|
557
582
|
|
558
|
-
gem install
|
559
|
-
|
560
|
-
If you cannot install `yajl-ruby` (JRuby?), you can install the `json`
|
561
|
-
gem and Resque will use it instead.
|
562
|
-
|
563
|
-
When problems arise, make sure you have the newest versions of the
|
564
|
-
`redis` and `redis-namespace` gems.
|
583
|
+
$ gem install bundler
|
584
|
+
$ bundle install
|
565
585
|
|
566
586
|
|
567
587
|
Installing Resque
|
@@ -575,7 +595,9 @@ First install the gem.
|
|
575
595
|
|
576
596
|
Next include it in your application.
|
577
597
|
|
578
|
-
|
598
|
+
``` ruby
|
599
|
+
require 'resque'
|
600
|
+
```
|
579
601
|
|
580
602
|
Now start your application:
|
581
603
|
|
@@ -586,8 +608,10 @@ That's it! You can now create Resque jobs from within your app.
|
|
586
608
|
To start a worker, create a Rakefile in your app's root (or add this
|
587
609
|
to an existing Rakefile):
|
588
610
|
|
589
|
-
|
590
|
-
|
611
|
+
``` ruby
|
612
|
+
require 'your/app'
|
613
|
+
require 'resque/tasks'
|
614
|
+
```
|
591
615
|
|
592
616
|
Now:
|
593
617
|
|
@@ -597,7 +621,7 @@ Alternately you can define a `resque:setup` hook in your Rakefile if you
|
|
597
621
|
don't want to load your app every time rake runs.
|
598
622
|
|
599
623
|
|
600
|
-
### In a Rails app, as a gem
|
624
|
+
### In a Rails 2.x app, as a gem
|
601
625
|
|
602
626
|
First install the gem.
|
603
627
|
|
@@ -616,7 +640,9 @@ That's it! You can now create Resque jobs from within your app.
|
|
616
640
|
|
617
641
|
To start a worker, add this to your Rakefile in `RAILS_ROOT`:
|
618
642
|
|
619
|
-
|
643
|
+
``` ruby
|
644
|
+
require 'resque/tasks'
|
645
|
+
```
|
620
646
|
|
621
647
|
Now:
|
622
648
|
|
@@ -626,7 +652,7 @@ Don't forget you can define a `resque:setup` hook in
|
|
626
652
|
`lib/tasks/whatever.rake` that loads the `environment` task every time.
|
627
653
|
|
628
654
|
|
629
|
-
### In a Rails app, as a plugin
|
655
|
+
### In a Rails 2.x app, as a plugin
|
630
656
|
|
631
657
|
$ ./script/plugin install git://github.com/defunkt/resque
|
632
658
|
|
@@ -641,6 +667,40 @@ Don't forget you can define a `resque:setup` hook in
|
|
641
667
|
`lib/tasks/whatever.rake` that loads the `environment` task every time.
|
642
668
|
|
643
669
|
|
670
|
+
### In a Rails 3 app, as a gem
|
671
|
+
|
672
|
+
First include it in your Gemfile.
|
673
|
+
|
674
|
+
$ cat Gemfile
|
675
|
+
...
|
676
|
+
gem 'resque'
|
677
|
+
...
|
678
|
+
|
679
|
+
Next install it with Bundler.
|
680
|
+
|
681
|
+
$ bundle install
|
682
|
+
|
683
|
+
Now start your application:
|
684
|
+
|
685
|
+
$ rails server
|
686
|
+
|
687
|
+
That's it! You can now create Resque jobs from within your app.
|
688
|
+
|
689
|
+
To start a worker, add this to a file in `lib/tasks` (ex:
|
690
|
+
`lib/tasks/resque.rake`):
|
691
|
+
|
692
|
+
``` ruby
|
693
|
+
require 'resque/tasks'
|
694
|
+
```
|
695
|
+
|
696
|
+
Now:
|
697
|
+
|
698
|
+
$ QUEUE=* rake environment resque:work
|
699
|
+
|
700
|
+
Don't forget you can define a `resque:setup` hook in
|
701
|
+
`lib/tasks/whatever.rake` that loads the `environment` task every time.
|
702
|
+
|
703
|
+
|
644
704
|
Configuration
|
645
705
|
-------------
|
646
706
|
|
@@ -669,11 +729,13 @@ Here's our `config/resque.yml`:
|
|
669
729
|
|
670
730
|
And our initializer:
|
671
731
|
|
672
|
-
|
673
|
-
|
732
|
+
``` ruby
|
733
|
+
rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
|
734
|
+
rails_env = ENV['RAILS_ENV'] || 'development'
|
674
735
|
|
675
|
-
|
676
|
-
|
736
|
+
resque_config = YAML.load_file(rails_root + '/config/resque.yml')
|
737
|
+
Resque.redis = resque_config[rails_env]
|
738
|
+
```
|
677
739
|
|
678
740
|
Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
|
679
741
|
this way we can tell our Sinatra app about the config file:
|
@@ -685,7 +747,9 @@ Now everyone is on the same page.
|
|
685
747
|
Also, you could disable jobs queueing by setting 'inline' attribute.
|
686
748
|
For example, if you want to run all jobs in the same process for cucumber, try:
|
687
749
|
|
688
|
-
|
750
|
+
``` ruby
|
751
|
+
Resque.inline = ENV['RAILS_ENV'] == "cucumber"
|
752
|
+
```
|
689
753
|
|
690
754
|
|
691
755
|
Plugins and Hooks
|
@@ -712,7 +776,9 @@ in your Redis server.
|
|
712
776
|
|
713
777
|
Simply use the `Resque.redis.namespace` accessor:
|
714
778
|
|
715
|
-
|
779
|
+
``` ruby
|
780
|
+
Resque.redis.namespace = "resque:GitHub"
|
781
|
+
```
|
716
782
|
|
717
783
|
We recommend sticking this in your initializer somewhere after Redis
|
718
784
|
is configured.
|
@@ -743,6 +809,13 @@ of charge. This is **not** used by GitHub in production, so please
|
|
743
809
|
send patches for any tweaks or improvements you can make to it.
|
744
810
|
|
745
811
|
|
812
|
+
Questions
|
813
|
+
---------
|
814
|
+
|
815
|
+
Please add them to the [FAQ](https://github.com/defunkt/resque/wiki/FAQ) or
|
816
|
+
ask on the Mailing List. The Mailing List is explained further below
|
817
|
+
|
818
|
+
|
746
819
|
Development
|
747
820
|
-----------
|
748
821
|
|
data/Rakefile
CHANGED
@@ -1,4 +1,8 @@
|
|
1
|
-
|
1
|
+
#
|
2
|
+
# Setup
|
3
|
+
#
|
4
|
+
|
5
|
+
load 'lib/tasks/redis.rake'
|
2
6
|
|
3
7
|
$LOAD_PATH.unshift 'lib'
|
4
8
|
require 'resque/tasks'
|
@@ -27,7 +31,7 @@ else
|
|
27
31
|
Rake::TestTask.new do |test|
|
28
32
|
test.libs << "test"
|
29
33
|
test.test_files = FileList['test/**/*_test.rb']
|
30
|
-
end
|
34
|
+
end
|
31
35
|
end
|
32
36
|
|
33
37
|
if command? :kicker
|
data/lib/resque.rb
CHANGED
@@ -1,11 +1,5 @@
|
|
1
1
|
require 'mongo'
|
2
2
|
|
3
|
-
begin
|
4
|
-
require 'yajl'
|
5
|
-
rescue LoadError
|
6
|
-
require 'json'
|
7
|
-
end
|
8
|
-
|
9
3
|
require 'resque/version'
|
10
4
|
|
11
5
|
require 'resque/errors'
|
@@ -173,6 +167,19 @@ module Resque
|
|
173
167
|
|
174
168
|
# Pushes a job onto a queue. Queue name should be a string and the
|
175
169
|
# item should be any JSON-able Ruby object.
|
170
|
+
#
|
171
|
+
# Resque works generally expect the `item` to be a hash with the following
|
172
|
+
# keys:
|
173
|
+
#
|
174
|
+
# class - The String name of the job to run.
|
175
|
+
# args - An Array of arguments to pass the job. Usually passed
|
176
|
+
# via `class.to_class.perform(*args)`.
|
177
|
+
#
|
178
|
+
# Example
|
179
|
+
#
|
180
|
+
# Resque.push('archive', :class => 'Archive', :args => [ 35, 'tar' ])
|
181
|
+
#
|
182
|
+
# Returns nothing
|
176
183
|
def push(queue, item)
|
177
184
|
watch_queue(queue)
|
178
185
|
mongo << { :queue => queue.to_s, :item => item , :date => Time.now }
|
@@ -259,6 +266,12 @@ module Resque
|
|
259
266
|
#
|
260
267
|
# This method is considered part of the `stable` API.
|
261
268
|
def enqueue(klass, *args)
|
269
|
+
# Perform before_enqueue hooks. Don't perform enqueue if any hook returns false
|
270
|
+
before_hooks = Plugin.before_enqueue_hooks(klass).collect do |hook|
|
271
|
+
klass.send(hook, *args)
|
272
|
+
end
|
273
|
+
return if before_hooks.any? { |result| result == false }
|
274
|
+
|
262
275
|
Job.create(queue_from_class(klass), klass, *args)
|
263
276
|
|
264
277
|
Plugin.after_enqueue_hooks(klass).each do |hook|
|
@@ -388,3 +401,4 @@ module Resque
|
|
388
401
|
end
|
389
402
|
|
390
403
|
end
|
404
|
+
|