resque-mongo 1.4.0 → 1.8.1
Sign up to get free protection for your applications and to get access to all the features.
- data/CONTRIBUTORS +24 -6
- data/HISTORY.md +65 -0
- data/README.markdown +34 -5
- data/Rakefile +1 -1
- data/bin/resque +2 -2
- data/bin/resque-web +6 -1
- data/deps.rip +2 -2
- data/docs/HOOKS.md +121 -0
- data/docs/PLUGINS.md +93 -0
- data/examples/demo/Rakefile +5 -0
- data/examples/monit/resque.monit +6 -0
- data/lib/resque.rb +94 -7
- data/lib/resque/errors.rb +3 -0
- data/lib/resque/failure.rb +3 -0
- data/lib/resque/failure/base.rb +3 -0
- data/lib/resque/failure/hoptoad.rb +29 -19
- data/lib/resque/failure/mongo.rb +10 -1
- data/lib/resque/helpers.rb +8 -2
- data/lib/resque/job.rb +107 -2
- data/lib/resque/plugin.rb +46 -0
- data/lib/resque/server.rb +30 -11
- data/lib/resque/server/public/ranger.js +50 -7
- data/lib/resque/server/public/style.css +8 -1
- data/lib/resque/server/test_helper.rb +19 -0
- data/lib/resque/server/views/failed.erb +17 -3
- data/lib/resque/server/views/key_sets.erb +20 -0
- data/lib/resque/server/views/{key.erb → key_string.erb} +2 -8
- data/lib/resque/server/views/queues.erb +5 -2
- data/lib/resque/server/views/stats.erb +2 -2
- data/lib/resque/server/views/workers.erb +1 -1
- data/lib/resque/server/views/working.erb +2 -0
- data/lib/resque/tasks.rb +1 -1
- data/lib/resque/version.rb +1 -1
- data/lib/resque/worker.rb +54 -15
- data/tasks/redis.rake +53 -29
- data/test/job_hooks_test.rb +302 -0
- data/test/job_plugins_test.rb +209 -0
- data/test/plugin_test.rb +116 -0
- data/test/resque-mongo_benchmark.rb +62 -0
- data/test/resque-web_test.rb +54 -0
- data/test/resque_test.rb +34 -0
- data/test/test_helper.rb +15 -0
- data/test/worker_test.rb +62 -2
- metadata +58 -23
data/CONTRIBUTORS
CHANGED
@@ -1,14 +1,32 @@
|
|
1
1
|
* Chris Wanstrath
|
2
|
+
* gravis
|
3
|
+
* scotttam
|
2
4
|
* John Barnette
|
3
|
-
*
|
5
|
+
* Mike Mangino
|
4
6
|
* Rob Hanlon
|
5
7
|
* Jason Amster
|
6
|
-
* jgeiger
|
7
8
|
* Aaron Quint
|
9
|
+
* Adam Cooke
|
10
|
+
* Ashley Martens
|
11
|
+
* Matt Duncan
|
12
|
+
* Michael Dwan
|
13
|
+
* Daniel Ceballos
|
14
|
+
* Roman Heinrich
|
15
|
+
* Thibaut Barrère
|
16
|
+
* jgeiger
|
17
|
+
* Simon Rozet
|
18
|
+
* Dave Hoover
|
19
|
+
* Christos Trochalakis
|
8
20
|
* Ben VandenBos
|
9
|
-
*
|
21
|
+
* snusnu
|
10
22
|
* Arthur Zapparoli
|
11
|
-
*
|
23
|
+
* Ben Marini
|
12
24
|
* Brian P O'Rourke
|
13
|
-
*
|
14
|
-
*
|
25
|
+
* Jim Remsik and Les Hill
|
26
|
+
* Karel Minarik
|
27
|
+
* Luc Castera
|
28
|
+
* Masatomo Nakano
|
29
|
+
* Matt Palmer
|
30
|
+
* PJ Hyett
|
31
|
+
* Roland Moriz
|
32
|
+
* malomalo
|
data/HISTORY.md
CHANGED
@@ -1,3 +1,68 @@
|
|
1
|
+
## 1.8.0 (2010-04-07)
|
2
|
+
|
3
|
+
* Jobs that never complete due to killed worker are now failed.
|
4
|
+
* Worker "working" state is now maintained by the parent, not the child.
|
5
|
+
* Stopped using deprecated redis.rb methods
|
6
|
+
* `Worker.working` race condition fixed
|
7
|
+
* `Worker#process` has been deprecated.
|
8
|
+
* Monit example fixed
|
9
|
+
* Redis::Client and Redis::Namespace can be passed to `Resque.redis=`
|
10
|
+
|
11
|
+
## 1.7.1 (2010-04-02)
|
12
|
+
|
13
|
+
* Bugfix: Make job hook execution order consistent
|
14
|
+
* Bugfix: stdout buffering in child process
|
15
|
+
|
16
|
+
## 1.7.0 (2010-03-31)
|
17
|
+
|
18
|
+
* Job hooks API. See docs/HOOKS.md.
|
19
|
+
* web: Hovering over dates shows a timestamp
|
20
|
+
* web: AJAXify retry action for failed jobs
|
21
|
+
* web bugfix: Fix pagination bug
|
22
|
+
|
23
|
+
## 1.6.1 (2010-03-25)
|
24
|
+
|
25
|
+
* Bugfix: Workers may not be clearing their state correctly on
|
26
|
+
shutdown
|
27
|
+
* Added example monit config.
|
28
|
+
* Exception class is now recorded when an error is raised in a
|
29
|
+
worker.
|
30
|
+
* web: Unit tests
|
31
|
+
* web: Show namespace in header and footer
|
32
|
+
* web: Remove a queue
|
33
|
+
* web: Retry failed jobs
|
34
|
+
|
35
|
+
## 1.6.0 (2010-03-09)
|
36
|
+
|
37
|
+
* Added `before_first_fork`, `before_fork`, and `after_fork` hooks.
|
38
|
+
* Hoptoad: Added server_environment config setting
|
39
|
+
* Hoptoad bugfix: Don't depend on RAILS_ROOT
|
40
|
+
* 1.8.6 compat fixes
|
41
|
+
|
42
|
+
## 1.5.2 (2010-03-03)
|
43
|
+
|
44
|
+
* Bugfix: JSON check was crazy.
|
45
|
+
|
46
|
+
## 1.5.1 (2010-03-03)
|
47
|
+
|
48
|
+
* `Job.destroy` and `Resque.dequeue` return the # of destroyed jobs.
|
49
|
+
* Hoptoad notifier improvements
|
50
|
+
* Specify the namespace with `resque-web` by passing `-N namespace`
|
51
|
+
* Bugfix: Don't crash when trying to parse invalid JSON.
|
52
|
+
* Bugfix: Non-standard namespace support
|
53
|
+
* Web: Red backgound for queue "failed" only shown if there are failed jobs.
|
54
|
+
* Web bugfix: Tabs highlight properly now
|
55
|
+
* Web bugfix: ZSET partial support in stats
|
56
|
+
* Web bugfix: Deleting failed jobs works again
|
57
|
+
* Web bugfix: Sets (or zsets, lists, etc) now paginate.
|
58
|
+
|
59
|
+
## 1.5.0 (2010-02-17)
|
60
|
+
|
61
|
+
* Version now included in procline, e.g. `resque-1.5.0: Message`
|
62
|
+
* Web bugfix: Ignore idle works in the "working" page
|
63
|
+
* Added `Resque::Job.destroy(queue, klass, *args)`
|
64
|
+
* Added `Resque.dequeue(klass, *args)`
|
65
|
+
|
1
66
|
## 1.4.0 (2010-02-11)
|
2
67
|
|
3
68
|
* Fallback when unable to bind QUIT and USR1 for Windows and JRuby.
|
data/README.markdown
CHANGED
@@ -7,14 +7,11 @@ monitoring sinatra app works except the "stats" panel, although there are
|
|
7
7
|
a lot of details that need to been taken care of.
|
8
8
|
|
9
9
|
Resque-mongo uses a fairly new feature of mongo, [findAndModify()][fnr].
|
10
|
-
findAndModify is
|
11
|
-
command's api might change a bit. You can use a [patched version
|
12
|
-
mongo-ruby-driver][fnr-ruby] that supports the current implementation.
|
10
|
+
findAndModify that is only supported by 0.20+ version ofthe ruby mongo driver.
|
13
11
|
|
14
12
|
Also, check your mongo version: 1.3.0 or higher
|
15
13
|
|
16
14
|
[fnr]: http://www.mongodb.org/display/DOCS/findandmodify+Command
|
17
|
-
[fnr-ruby]: http://github.com/ctrochalakis/mongo-ruby-driver/tree/find_replace
|
18
15
|
|
19
16
|
Resque
|
20
17
|
======
|
@@ -428,6 +425,10 @@ the script as the final argument:
|
|
428
425
|
|
429
426
|
$ resque-web -p 8282 rails_root/config/initializers/resque.rb
|
430
427
|
|
428
|
+
You can also set the namespace directly using `resque-web`:
|
429
|
+
|
430
|
+
$ resque-web -p 8282 -N myapp
|
431
|
+
|
431
432
|
### Passenger
|
432
433
|
|
433
434
|
Using Passenger? Resque ships with a `config.ru` you can use. See
|
@@ -499,6 +500,8 @@ tool that's best for your app.
|
|
499
500
|
Installing Redis
|
500
501
|
----------------
|
501
502
|
|
503
|
+
Resque requires Redis 0.900 or higher.
|
504
|
+
|
502
505
|
Resque uses Redis' lists for its queues. It also stores worker state
|
503
506
|
data in Redis.
|
504
507
|
|
@@ -543,6 +546,9 @@ Resque Dependencies
|
|
543
546
|
If you cannot install `yajl-ruby` (JRuby?), you can install the `json`
|
544
547
|
gem and Resque will use it instead.
|
545
548
|
|
549
|
+
When problems arise, make sure you have the newest versions of the
|
550
|
+
`redis` and `redis-namespace` gems.
|
551
|
+
|
546
552
|
|
547
553
|
Installing Resque
|
548
554
|
-----------------
|
@@ -663,6 +669,17 @@ this way we can tell our Sinatra app about the config file:
|
|
663
669
|
Now everyone is on the same page.
|
664
670
|
|
665
671
|
|
672
|
+
Plugins and Hooks
|
673
|
+
-----------------
|
674
|
+
|
675
|
+
For a list of available plugins see
|
676
|
+
<http://wiki.github.com/defunkt/resque/plugins>.
|
677
|
+
|
678
|
+
If you'd like to write your own plugin, or want to customize Resque
|
679
|
+
using hooks (such as `Resque.after_fork`), see
|
680
|
+
[docs/HOOKS.md](http://github.com/defunkt/resque/blob/master/HOOKS.md).
|
681
|
+
|
682
|
+
|
666
683
|
Namespaces
|
667
684
|
----------
|
668
685
|
|
@@ -694,10 +711,18 @@ Try it out by looking at the README, found at `examples/demo/README.markdown`.
|
|
694
711
|
Monitoring
|
695
712
|
----------
|
696
713
|
|
714
|
+
### god
|
715
|
+
|
697
716
|
If you're using god to monitor Resque, we have provided example
|
698
717
|
configs in `examples/god/`. One is for starting / stopping workers,
|
699
718
|
the other is for killing workers that have been running too long.
|
700
719
|
|
720
|
+
### monit
|
721
|
+
|
722
|
+
If you're using monit, `examples/monit/resque.monit` is provided free
|
723
|
+
of charge. This is **not** used by GitHub in production, so please
|
724
|
+
send patches for any tweaks or improvements you can make to it.
|
725
|
+
|
701
726
|
|
702
727
|
Development
|
703
728
|
-----------
|
@@ -743,13 +768,16 @@ Once you've made your great commits:
|
|
743
768
|
4. Create an [Issue][2] with a link to your branch
|
744
769
|
5. That's it!
|
745
770
|
|
771
|
+
You might want to checkout our [Contributing][cb] wiki page for information
|
772
|
+
on coding standards, new features, etc.
|
773
|
+
|
746
774
|
|
747
775
|
Mailing List
|
748
776
|
------------
|
749
777
|
|
750
778
|
To join the list simply send an email to <resque@librelist.com>. This
|
751
779
|
will subscribe you and send you information about your subscription,
|
752
|
-
|
780
|
+
including unsubscribe information.
|
753
781
|
|
754
782
|
The archive can be found at <http://librelist.com/browser/>.
|
755
783
|
|
@@ -778,3 +806,4 @@ Chris Wanstrath :: chris@ozmm.org :: @defunkt
|
|
778
806
|
[2]: http://github.com/defunkt/resque/issues
|
779
807
|
[sv]: http://semver.org/
|
780
808
|
[rs]: http://github.com/defunkt/redis-namespace
|
809
|
+
[cb]: http://wiki.github.com/defunkt/resque/contributing
|
data/Rakefile
CHANGED
@@ -38,7 +38,7 @@ begin
|
|
38
38
|
gemspec.authors = ["Christos Trochalakis"]
|
39
39
|
gemspec.version = Resque::Version
|
40
40
|
|
41
|
-
gemspec.add_dependency "mongo"
|
41
|
+
gemspec.add_dependency "mongo", ">=0.20"
|
42
42
|
gemspec.add_dependency "vegas", ">=0.1.2"
|
43
43
|
gemspec.add_dependency "sinatra", ">=0.9.2"
|
44
44
|
gemspec.add_development_dependency "jeweler"
|
data/bin/resque
CHANGED
@@ -35,7 +35,7 @@ def list
|
|
35
35
|
end
|
36
36
|
|
37
37
|
if (i = ARGV.index('-r')) && ARGV[i+1]
|
38
|
-
Resque.
|
38
|
+
Resque.mongo = ARGV[i+1]
|
39
39
|
ARGV.delete_at(i)
|
40
40
|
ARGV.delete_at(i+1)
|
41
41
|
end
|
@@ -48,7 +48,7 @@ when 'remove'
|
|
48
48
|
when 'list'
|
49
49
|
list
|
50
50
|
else
|
51
|
-
puts "Usage: resque [-r
|
51
|
+
puts "Usage: resque [-r mongo_host:mongo_port] COMMAND [option]"
|
52
52
|
puts
|
53
53
|
puts "Commands:"
|
54
54
|
puts " remove WORKER Removes a worker"
|
data/bin/resque-web
CHANGED
@@ -15,4 +15,9 @@ Vegas::Runner.new(Resque::Server, 'resque-web', {
|
|
15
15
|
path = (ENV['RESQUECONFIG'] || v.args.first)
|
16
16
|
load path.to_s.strip if path
|
17
17
|
}
|
18
|
-
})
|
18
|
+
}) do |runner, opts, app|
|
19
|
+
opts.on('-N NAMESPACE', "--namespace NAMESPACE", "set the Redis namespace") {|namespace|
|
20
|
+
runner.logger.info "Using Redis namespace '#{namespace}'"
|
21
|
+
Resque.redis.namespace = namespace
|
22
|
+
}
|
23
|
+
end
|
data/deps.rip
CHANGED
@@ -1,8 +1,8 @@
|
|
1
|
-
git://github.com/ctrochalakis/mongo-ruby-driver find_replace
|
2
1
|
mongo
|
3
|
-
|
2
|
+
bson_ext
|
4
3
|
git://github.com/brianmario/yajl-ruby.git 0.6.3
|
5
4
|
git://github.com/sinatra/sinatra.git 0.9.4
|
6
5
|
git://github.com/rack/rack.git 1.0
|
7
6
|
git://github.com/quirkey/vegas.git v0.1.2
|
7
|
+
git://github.com/brynary/rack-test.git v0.5.3
|
8
8
|
rake
|
data/docs/HOOKS.md
ADDED
@@ -0,0 +1,121 @@
|
|
1
|
+
Resque Hooks
|
2
|
+
============
|
3
|
+
|
4
|
+
You can customize Resque or write plugins using its hook API. In many
|
5
|
+
cases you can use a hook rather than mess with Resque's internals.
|
6
|
+
|
7
|
+
For a list of available plugins see
|
8
|
+
<http://wiki.github.com/defunkt/resque/plugins>.
|
9
|
+
|
10
|
+
|
11
|
+
Worker Hooks
|
12
|
+
------------
|
13
|
+
|
14
|
+
If you wish to have a Proc called before the worker forks for the
|
15
|
+
first time, you can add it in the initializer like so:
|
16
|
+
|
17
|
+
Resque.before_first_fork do
|
18
|
+
puts "Call me once before the worker forks the first time"
|
19
|
+
end
|
20
|
+
|
21
|
+
You can also run a hook before _every_ fork:
|
22
|
+
|
23
|
+
Resque.before_fork do |job|
|
24
|
+
puts "Call me before the worker forks"
|
25
|
+
end
|
26
|
+
|
27
|
+
The `before_fork` hook will be run in the **parent** process. So, be
|
28
|
+
careful - any changes you make will be permanent for the lifespan of
|
29
|
+
the worker.
|
30
|
+
|
31
|
+
And after forking:
|
32
|
+
|
33
|
+
Resque.after_fork do |job|
|
34
|
+
puts "Call me after the worker forks"
|
35
|
+
end
|
36
|
+
|
37
|
+
The `after_fork` hook will be run in the child process and is passed
|
38
|
+
the current job. Any changes you make, therefor, will only live as
|
39
|
+
long as the job currently being processes.
|
40
|
+
|
41
|
+
All worker hooks can also be set using a setter, e.g.
|
42
|
+
|
43
|
+
Resque.after_fork = proc { puts "called" }
|
44
|
+
|
45
|
+
|
46
|
+
Job Hooks
|
47
|
+
---------
|
48
|
+
|
49
|
+
Plugins can utilize job hooks to provide additional behavior. A job
|
50
|
+
hook is a method name in the following format:
|
51
|
+
|
52
|
+
HOOKNAME_IDENTIFIER
|
53
|
+
|
54
|
+
For example, a `before_perform` hook which adds locking may be defined
|
55
|
+
like this:
|
56
|
+
|
57
|
+
def before_perform_with_lock(*args)
|
58
|
+
set_lock!
|
59
|
+
end
|
60
|
+
|
61
|
+
Once this hook is made available to your job (either by way of
|
62
|
+
inheritence or `extend`), it will be run before the job's `perform`
|
63
|
+
method is called. Hooks of each type are executed in alphabetical order,
|
64
|
+
so `before_perform_a` will always be executed before `before_perform_b`.
|
65
|
+
An unnamed hook (`before_perform`) will be executed first.
|
66
|
+
|
67
|
+
The available hooks are:
|
68
|
+
|
69
|
+
* `before_perform`: Called with the job args before perform. If it raises
|
70
|
+
`Resque::Job::DontPerform`, the job is aborted. If other exceptions
|
71
|
+
are raised, they will be propagated up the the `Resque::Failure`
|
72
|
+
backend.
|
73
|
+
|
74
|
+
* `after_perform`: Called with the job args after it performs. Uncaught
|
75
|
+
exceptions will propagate up to the `Resque::Failure` backend.
|
76
|
+
|
77
|
+
* `around_perform`: Called with the job args. It is expected to yield in order
|
78
|
+
to perform the job (but is not required to do so). It may handle exceptions
|
79
|
+
thrown by `perform`, but any that are not caught will propagate up to the
|
80
|
+
`Resque::Failure` backend.
|
81
|
+
|
82
|
+
* `on_failure`: Called with the exception and job args if any exception occurs
|
83
|
+
while performing the job (or hooks).
|
84
|
+
|
85
|
+
Hooks are easily implemented with superclasses or modules. A superclass could
|
86
|
+
look something like this.
|
87
|
+
|
88
|
+
class LoggedJob
|
89
|
+
def self.before_perform_log_job(*args)
|
90
|
+
Logger.info "About to perform #{self} with #{args.inspect}"
|
91
|
+
end
|
92
|
+
end
|
93
|
+
|
94
|
+
class MyJob < LoggedJob
|
95
|
+
def self.perform(*args)
|
96
|
+
...
|
97
|
+
end
|
98
|
+
end
|
99
|
+
|
100
|
+
Modules are even better because jobs can use many of them.
|
101
|
+
|
102
|
+
module LoggedJob
|
103
|
+
def before_perform_log_job(*args)
|
104
|
+
Logger.info "About to perform #{self} with #{args.inspect}"
|
105
|
+
end
|
106
|
+
end
|
107
|
+
|
108
|
+
module RetriedJob
|
109
|
+
def on_failure_retry(e, *args)
|
110
|
+
Logger.info "Performing #{self} caused an exception (#{e}). Retrying..."
|
111
|
+
Resque.enqueue self, *args
|
112
|
+
end
|
113
|
+
end
|
114
|
+
|
115
|
+
class MyJob
|
116
|
+
extend LoggedJob
|
117
|
+
extend RetriedJob
|
118
|
+
def self.perform(*args)
|
119
|
+
...
|
120
|
+
end
|
121
|
+
end
|
data/docs/PLUGINS.md
ADDED
@@ -0,0 +1,93 @@
|
|
1
|
+
Resque Plugins
|
2
|
+
==============
|
3
|
+
|
4
|
+
Resque encourages plugin development. For a list of available plugins,
|
5
|
+
please see <http://wiki.github.com/defunkt/resque/plugins>.
|
6
|
+
|
7
|
+
The `docs/HOOKS.md` file included with Resque documents the available
|
8
|
+
hooks you can use to add or change Resque functionality. This document
|
9
|
+
describes best practice for plugins themselves.
|
10
|
+
|
11
|
+
|
12
|
+
Version
|
13
|
+
-------
|
14
|
+
|
15
|
+
Plugins should declare the major.minor version of Resque they are
|
16
|
+
known to work with explicitly in their README.
|
17
|
+
|
18
|
+
For example, if your plugin depends on features in Resque 2.1, please
|
19
|
+
list "Depends on Resque 2.1" very prominently near the beginning of
|
20
|
+
your README.
|
21
|
+
|
22
|
+
Because Resque uses [Semantic Versioning][sv], you can safely make the
|
23
|
+
following assumptions:
|
24
|
+
|
25
|
+
* Your plugin will work with 2.2, 2.3, etc - no methods will be
|
26
|
+
removed or changed, only added.
|
27
|
+
* Your plugin might not work with 3.0+, as APIs may change or be
|
28
|
+
removed.
|
29
|
+
|
30
|
+
|
31
|
+
Namespace
|
32
|
+
---------
|
33
|
+
|
34
|
+
All plugins should live under the `Resque::Plugins` module to avoid
|
35
|
+
clashing with first class Resque constants or other Ruby libraries.
|
36
|
+
|
37
|
+
Good:
|
38
|
+
|
39
|
+
* Resque::Plugins::Lock
|
40
|
+
* Resque::Plugins::FastRetry
|
41
|
+
|
42
|
+
Bad:
|
43
|
+
|
44
|
+
* Resque::Lock
|
45
|
+
* ResqueQueue
|
46
|
+
|
47
|
+
|
48
|
+
Gem Name
|
49
|
+
--------
|
50
|
+
|
51
|
+
Gem names should be in the format of `resque-FEATURE`, where `FEATURE`
|
52
|
+
succinctly describes the feature your plugin adds to Resque.
|
53
|
+
|
54
|
+
Good:
|
55
|
+
|
56
|
+
* resque-status
|
57
|
+
* resque-scheduler
|
58
|
+
|
59
|
+
Bad:
|
60
|
+
|
61
|
+
* multi-queue
|
62
|
+
* defunkt-resque-lock
|
63
|
+
|
64
|
+
|
65
|
+
Hooks
|
66
|
+
-----
|
67
|
+
|
68
|
+
Job hook names should be namespaced to work properly.
|
69
|
+
|
70
|
+
Good:
|
71
|
+
|
72
|
+
* before_perform_lock
|
73
|
+
* around_perform_check_status
|
74
|
+
|
75
|
+
Bad:
|
76
|
+
|
77
|
+
* before_perform
|
78
|
+
* on_failure
|
79
|
+
|
80
|
+
|
81
|
+
Lint
|
82
|
+
----
|
83
|
+
|
84
|
+
Plugins should test compliance to this document using the
|
85
|
+
`Resque::Plugin.lint` method.
|
86
|
+
|
87
|
+
For example:
|
88
|
+
|
89
|
+
assert_nothing_raised do
|
90
|
+
Resque::Plugin.lint(Resque::Plugins::Lock)
|
91
|
+
end
|
92
|
+
|
93
|
+
[sv]: http://semver.org/
|