stalking 0.0.1
Sign up to get free protection for your applications and to get access to all the features.
- data/.gitignore +17 -0
- data/Gemfile +4 -0
- data/LICENSE.txt +22 -0
- data/README.md +187 -0
- data/Rakefile +9 -0
- data/bin/stalking +11 -0
- data/lib/stalking.rb +10 -0
- data/lib/stalking/consumer.rb +117 -0
- data/lib/stalking/handlers.rb +31 -0
- data/lib/stalking/producer.rb +74 -0
- data/lib/stalking/version.rb +3 -0
- data/stalking.gemspec +24 -0
- data/test/stalking/beanstalk.rb +136 -0
- data/test/stalking/consumer_test.rb +208 -0
- data/test/stalking/producer_test.rb +86 -0
- data/test/stalking/test_logger.rb +13 -0
- metadata +111 -0
data/.gitignore
ADDED
data/Gemfile
ADDED
data/LICENSE.txt
ADDED
@@ -0,0 +1,22 @@
|
|
1
|
+
Copyright (c) 2012 Benjamin Vetter
|
2
|
+
|
3
|
+
MIT License
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
6
|
+
a copy of this software and associated documentation files (the
|
7
|
+
"Software"), to deal in the Software without restriction, including
|
8
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
9
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
10
|
+
permit persons to whom the Software is furnished to do so, subject to
|
11
|
+
the following conditions:
|
12
|
+
|
13
|
+
The above copyright notice and this permission notice shall be
|
14
|
+
included in all copies or substantial portions of the Software.
|
15
|
+
|
16
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
17
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
18
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
19
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
20
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
21
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
22
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,187 @@
|
|
1
|
+
|
2
|
+
# Stalking
|
3
|
+
|
4
|
+
Asynchronous job processing using the beanstalkd queue server with automatic
|
5
|
+
reconnects and failover for your ruby projects.
|
6
|
+
|
7
|
+
Background jobs are essential to every project these days and the tools should
|
8
|
+
be as reliable as possible.
|
9
|
+
|
10
|
+
Beanstalkd is perfectly suited for background processing and the stalker gem
|
11
|
+
has long been my favorite to integrate beanstalkd into my projects.
|
12
|
+
Unfortunately, stalker does not provide automatic reconnects and failover.
|
13
|
+
|
14
|
+
The DSL provided by stalking is very similar to the DSL of stalker with only
|
15
|
+
minimal changes driven by design decisions and its additional feature set.
|
16
|
+
|
17
|
+
## Installation
|
18
|
+
|
19
|
+
Add this line to your application's Gemfile:
|
20
|
+
|
21
|
+
gem 'stalking', :git => 'git://github.com/mrkamel/stalking.git'
|
22
|
+
|
23
|
+
And then execute:
|
24
|
+
|
25
|
+
$ bundle
|
26
|
+
|
27
|
+
## Usage
|
28
|
+
|
29
|
+
Stalking provides two modes of operation, a producer and a consumer mode.
|
30
|
+
|
31
|
+
Enqueuing new jobs is as easy as:
|
32
|
+
|
33
|
+
```ruby
|
34
|
+
Stalking::Producer.new.enqueue "name", "key" => "value"
|
35
|
+
```
|
36
|
+
|
37
|
+
This will try to connect to a beanstalkd instance running on `localhost:11300`
|
38
|
+
to enqueue a new job named `job` with the provided arguments. The arguments
|
39
|
+
will be serialized using JSON before being enqueued.
|
40
|
+
|
41
|
+
To specify a `priority`, `ttr` or `delay`, please use:
|
42
|
+
|
43
|
+
```ruby
|
44
|
+
Stalking::Producer.new.enqueue "name", { "key" => "value" }, :ttr => 360, :delay => 60, :pri => 100
|
45
|
+
```
|
46
|
+
|
47
|
+
You can of course use multiple beanstalkd instances, by using:
|
48
|
+
|
49
|
+
```ruby
|
50
|
+
producer = Stalking::Producer.new(:servers => ["192.168.1.100:11300", "192.168.1.101"])
|
51
|
+
producer.enqueue "name", "key" => "value"
|
52
|
+
```
|
53
|
+
|
54
|
+
Here, everytime you enqueue a new job, stalking will randomly choose a server
|
55
|
+
from your list to send the job to. Thus, you can e.g. create a producer for
|
56
|
+
local jobs (like e.g. for local background image processing) and another
|
57
|
+
producer for global jobs:
|
58
|
+
|
59
|
+
```ruby
|
60
|
+
global_queue = Stalking::Producer.new(:servers => ["192.168.1.100:11300", "192.168.1.101"])
|
61
|
+
local_queue = Stalking::Producer.new
|
62
|
+
```
|
63
|
+
|
64
|
+
If a server from your list is down or unreachable, stalking will use another
|
65
|
+
one from your list. Only in case all your servers are down, `enqueue` will
|
66
|
+
return false. However, please carefully read the `Caveats` section.
|
67
|
+
|
68
|
+
You can even set the `ttr`, `priority` and `delay` options
|
69
|
+
on a per-producer base:
|
70
|
+
|
71
|
+
```ruby
|
72
|
+
producer = Stalking::Producer.new(:pri => 100, :delay => 60, :ttr => 360)
|
73
|
+
```
|
74
|
+
|
75
|
+
To set an upper limit for servers to be tried out for queuing use the `tries`
|
76
|
+
option:
|
77
|
+
|
78
|
+
```ruby
|
79
|
+
producer = Stalking::Producer.new(:servers => [..., ..., ...], :tries => 2)
|
80
|
+
```
|
81
|
+
|
82
|
+
To consume jobs in a background worker, please use the following DSL, which is
|
83
|
+
quite similar to stalker's DSL:
|
84
|
+
|
85
|
+
```ruby
|
86
|
+
Stalking::Consumer.new do
|
87
|
+
job "name" do |args|
|
88
|
+
# Do something
|
89
|
+
end
|
90
|
+
|
91
|
+
job "another" do |args|
|
92
|
+
# Do something
|
93
|
+
end
|
94
|
+
|
95
|
+
...
|
96
|
+
end
|
97
|
+
```
|
98
|
+
|
99
|
+
After being specified, the consumer will immediately block and wait to consume
|
100
|
+
jobs from beanstalkd.
|
101
|
+
|
102
|
+
Additionally, you can specify before, after and error hooks:
|
103
|
+
|
104
|
+
```ruby
|
105
|
+
Stalking::Consumer.new do
|
106
|
+
before do |name, args|
|
107
|
+
# Executed before a job runs
|
108
|
+
end
|
109
|
+
|
110
|
+
after do |name, args|
|
111
|
+
# Executed after a job has run
|
112
|
+
end
|
113
|
+
|
114
|
+
error do |e, name, args|
|
115
|
+
# Executed when job exited abnormally
|
116
|
+
end
|
117
|
+
|
118
|
+
job "name" do |args|
|
119
|
+
# The job itself
|
120
|
+
end
|
121
|
+
end
|
122
|
+
```
|
123
|
+
|
124
|
+
If you save this file to e.g. `jobs.rb`, you can execute your worker like:
|
125
|
+
|
126
|
+
```
|
127
|
+
$ stalking jobs
|
128
|
+
```
|
129
|
+
|
130
|
+
Again, by default stalking connects to localhost:11300, but you can specify a
|
131
|
+
pool of servers to consume jobs from:
|
132
|
+
|
133
|
+
```ruby
|
134
|
+
Stalking::Consumer.new(:servers => ["192.168.1.100:11300", "192.168.1.101:11300"]) do
|
135
|
+
...
|
136
|
+
end
|
137
|
+
```
|
138
|
+
|
139
|
+
Please note that, due to the implementation of beanstalk-client, the consumer
|
140
|
+
requires all pool servers to be up. Otherwise, the consumer will enter a
|
141
|
+
reconnect loop, until all servers are up again. Thus, stalking will try to
|
142
|
+
reconnect to the servers when a connection is lost and it's recommended for
|
143
|
+
consumers to connect to a single server, usually.
|
144
|
+
|
145
|
+
That should be it.
|
146
|
+
|
147
|
+
## Caveats
|
148
|
+
|
149
|
+
Having producers which try to reconnect and/or try to connect to other servers
|
150
|
+
when a single server is down, of course require some time for the error
|
151
|
+
detection and processing. This usually should not be a big issue, except there
|
152
|
+
is some temporary lag between the producer and the beanstalkd server. The
|
153
|
+
producer then has to wait for a timeout and while the producer is waiting for
|
154
|
+
the timeout it will block your application. The default timeout is 250
|
155
|
+
milliseconds, but you can change the timeout via:
|
156
|
+
|
157
|
+
```ruby
|
158
|
+
Stalking::Producer.new :timeout => 0.1 # Set timeout to 100 milliseconds
|
159
|
+
```
|
160
|
+
|
161
|
+
The optimal timeout for your setup depends on your network connection, the
|
162
|
+
distance of your servers, etc. You should experiment a bit with it if you
|
163
|
+
experience some kind of problems. However, i don't emphasize to use stalking
|
164
|
+
producers which communicate with beanstalkd servers over WAN connections.
|
165
|
+
Only local, data-center connections to beanstalkd servers should be used.
|
166
|
+
|
167
|
+
# Error Logging
|
168
|
+
|
169
|
+
To log connection errors, you can provide a logger to the consumers as well
|
170
|
+
as producers:
|
171
|
+
|
172
|
+
```ruby
|
173
|
+
Stalking::Producer.new :logger => ...
|
174
|
+
Stalking::Consumer.new :logger => ...
|
175
|
+
```
|
176
|
+
|
177
|
+
You can hand out any kind of logging object, which responds to an `error`
|
178
|
+
method and takes a string error message.
|
179
|
+
|
180
|
+
## Contributing
|
181
|
+
|
182
|
+
1. Fork it
|
183
|
+
2. Create your feature branch (`git checkout -b my-new-feature`)
|
184
|
+
3. Commit your changes (`git commit -am 'Add some feature'`)
|
185
|
+
4. Push to the branch (`git push origin my-new-feature`)
|
186
|
+
5. Create new Pull Request
|
187
|
+
|
data/Rakefile
ADDED
data/bin/stalking
ADDED
data/lib/stalking.rb
ADDED
@@ -0,0 +1,117 @@
|
|
1
|
+
|
2
|
+
module Stalking
|
3
|
+
class Consumer
|
4
|
+
def initialize(options = {}, &block)
|
5
|
+
@servers = options[:servers] || ["localhost:11300"]
|
6
|
+
|
7
|
+
@logger = options[:logger]
|
8
|
+
|
9
|
+
@handlers = Handlers.new
|
10
|
+
|
11
|
+
if block_given?
|
12
|
+
@handlers.instance_eval &block
|
13
|
+
|
14
|
+
work
|
15
|
+
end
|
16
|
+
end
|
17
|
+
|
18
|
+
def handle(handlers, name, args)
|
19
|
+
handlers.each do |handler|
|
20
|
+
handler.call name, args
|
21
|
+
end
|
22
|
+
end
|
23
|
+
|
24
|
+
def handle_error(e, name, args)
|
25
|
+
@handlers.error_handlers.each do |handler|
|
26
|
+
handler.call e, name, args
|
27
|
+
end
|
28
|
+
end
|
29
|
+
|
30
|
+
private
|
31
|
+
|
32
|
+
def work
|
33
|
+
setup
|
34
|
+
|
35
|
+
begin
|
36
|
+
loop do
|
37
|
+
job = connection.reserve
|
38
|
+
|
39
|
+
exit if @quit
|
40
|
+
|
41
|
+
busy do
|
42
|
+
name, args = JSON.parse(job.body)
|
43
|
+
|
44
|
+
if handler = @handlers.job_handlers[name]
|
45
|
+
begin
|
46
|
+
Timeout::timeout(job.ttr - 1) do
|
47
|
+
handle @handlers.before_handlers, name, args
|
48
|
+
|
49
|
+
handler.call args
|
50
|
+
|
51
|
+
handle @handlers.after_handlers, name, args
|
52
|
+
end
|
53
|
+
rescue Beanstalk::NotConnected => e
|
54
|
+
raise e # Re-raise
|
55
|
+
rescue Timeout::Error, StandardError => e
|
56
|
+
handle_error e, name, args
|
57
|
+
end
|
58
|
+
end
|
59
|
+
end
|
60
|
+
|
61
|
+
job.delete
|
62
|
+
end
|
63
|
+
rescue Beanstalk::NotConnected => e
|
64
|
+
log_error e
|
65
|
+
|
66
|
+
sleep 1
|
67
|
+
|
68
|
+
reconnect
|
69
|
+
|
70
|
+
retry
|
71
|
+
end
|
72
|
+
end
|
73
|
+
|
74
|
+
def setup
|
75
|
+
trap "QUIT" do
|
76
|
+
exit unless @busy
|
77
|
+
|
78
|
+
@quit = true
|
79
|
+
end
|
80
|
+
end
|
81
|
+
|
82
|
+
def busy
|
83
|
+
@busy = true
|
84
|
+
|
85
|
+
yield
|
86
|
+
ensure
|
87
|
+
@busy = false
|
88
|
+
end
|
89
|
+
|
90
|
+
def connect
|
91
|
+
@connection ||= Beanstalk::Pool.new(@servers)
|
92
|
+
|
93
|
+
@handlers.job_handlers.keys.each { |name| @connection.watch name }
|
94
|
+
|
95
|
+
@connection
|
96
|
+
rescue Beanstalk::NotConnected => e
|
97
|
+
log_error e
|
98
|
+
|
99
|
+
sleep 1
|
100
|
+
|
101
|
+
retry
|
102
|
+
end
|
103
|
+
|
104
|
+
alias :connection :connect
|
105
|
+
|
106
|
+
def reconnect
|
107
|
+
@connection = nil
|
108
|
+
|
109
|
+
connect
|
110
|
+
end
|
111
|
+
|
112
|
+
def log_error(e)
|
113
|
+
@logger.error("Stalking error: #{e.message}") if @logger
|
114
|
+
end
|
115
|
+
end
|
116
|
+
end
|
117
|
+
|
@@ -0,0 +1,31 @@
|
|
1
|
+
|
2
|
+
module Stalking
|
3
|
+
class Handlers
|
4
|
+
attr_reader :before_handlers, :after_handlers, :error_handlers, :job_handlers
|
5
|
+
|
6
|
+
def initialize
|
7
|
+
@before_handlers = []
|
8
|
+
@after_handlers = []
|
9
|
+
@error_handlers = []
|
10
|
+
|
11
|
+
@job_handlers = {}
|
12
|
+
end
|
13
|
+
|
14
|
+
def before(&block)
|
15
|
+
@before_handlers.push block
|
16
|
+
end
|
17
|
+
|
18
|
+
def after(&block)
|
19
|
+
@after_handlers.push block
|
20
|
+
end
|
21
|
+
|
22
|
+
def error(&block)
|
23
|
+
@error_handlers.push block
|
24
|
+
end
|
25
|
+
|
26
|
+
def job(name, &block)
|
27
|
+
@job_handlers[name] = block
|
28
|
+
end
|
29
|
+
end
|
30
|
+
end
|
31
|
+
|
@@ -0,0 +1,74 @@
|
|
1
|
+
|
2
|
+
module Stalking
|
3
|
+
class Producer
|
4
|
+
def initialize(options = {}, &block)
|
5
|
+
@timeout = options[:timeout] || 0.25
|
6
|
+
|
7
|
+
@pri = options[:pri] || 65536
|
8
|
+
@delay = options[:delay] || 0
|
9
|
+
@ttr = options[:ttr] || 120
|
10
|
+
|
11
|
+
@logger = options[:logger]
|
12
|
+
|
13
|
+
@servers = options[:servers] || ["localhost:11300"]
|
14
|
+
|
15
|
+
@tries = options[:tries] || @servers.size
|
16
|
+
|
17
|
+
@connections = {}
|
18
|
+
end
|
19
|
+
|
20
|
+
def enqueue(name, args = {}, options = {})
|
21
|
+
# Send the job to a random server.
|
22
|
+
|
23
|
+
@servers.shuffle.first(@tries).each do |server|
|
24
|
+
begin
|
25
|
+
Timeout::timeout(@timeout) do
|
26
|
+
@connections[server] ||= Beanstalk::Pool.new([server])
|
27
|
+
|
28
|
+
enq @connections[server], name, args, options
|
29
|
+
end
|
30
|
+
|
31
|
+
return true
|
32
|
+
rescue Beanstalk::NotConnected
|
33
|
+
# Connecting to the beanstalk server has failed.
|
34
|
+
# Let's try to reconnect and enqueue afterwards.
|
35
|
+
|
36
|
+
begin
|
37
|
+
Timeout::timeout(@timeout) do
|
38
|
+
@connections[server] = Beanstalk::Pool.new([server])
|
39
|
+
|
40
|
+
enq @connections[server], name, args, options
|
41
|
+
end
|
42
|
+
|
43
|
+
return true
|
44
|
+
rescue Beanstalk::NotConnected, Timeout::Error, StandardError => e
|
45
|
+
log_error e
|
46
|
+
end
|
47
|
+
rescue Timeout::Error, StandardError => e
|
48
|
+
log_error e
|
49
|
+
end
|
50
|
+
end
|
51
|
+
|
52
|
+
# All servers are currently unavailable.
|
53
|
+
# Enqueuing the job has failed.
|
54
|
+
|
55
|
+
false
|
56
|
+
end
|
57
|
+
|
58
|
+
private
|
59
|
+
|
60
|
+
def enq(connection, name, args = {}, options = {})
|
61
|
+
pri = options[:pri] || @pri
|
62
|
+
delay = options[:delay] || @delay
|
63
|
+
ttr = options[:ttr] || @ttr
|
64
|
+
|
65
|
+
connection.use name
|
66
|
+
connection.put [name, args].to_json, pri, delay, ttr
|
67
|
+
end
|
68
|
+
|
69
|
+
def log_error(e)
|
70
|
+
@logger.error("Stalking error: #{e.message}") if @logger
|
71
|
+
end
|
72
|
+
end
|
73
|
+
end
|
74
|
+
|
data/stalking.gemspec
ADDED
@@ -0,0 +1,24 @@
|
|
1
|
+
# -*- encoding: utf-8 -*-
|
2
|
+
$:.push File.expand_path("../lib", __FILE__)
|
3
|
+
require "stalking/version"
|
4
|
+
|
5
|
+
Gem::Specification.new do |s|
|
6
|
+
s.name = "stalking"
|
7
|
+
s.version = Stalking::VERSION
|
8
|
+
s.authors = ["Benjamin Vetter"]
|
9
|
+
s.email = ["vetter@flakks.com"]
|
10
|
+
s.homepage = "https://github.com/mrkamel/stalking"
|
11
|
+
s.summary = %q{Asynchronous jobs using beanstalkd with automatic reconnects and failover}
|
12
|
+
s.description = %q{Asynchronous job processing using the beanstalkd queue server with automatic reconnects and failover}
|
13
|
+
|
14
|
+
s.files = `git ls-files`.split("\n")
|
15
|
+
s.test_files = `git ls-files -- {test,spec,features}/*`.split("\n")
|
16
|
+
s.executables = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
|
17
|
+
s.require_paths = ["lib"]
|
18
|
+
|
19
|
+
s.add_development_dependency "rake"
|
20
|
+
|
21
|
+
s.add_dependency "beanstalk-client"
|
22
|
+
s.add_dependency "json_pure"
|
23
|
+
end
|
24
|
+
|
@@ -0,0 +1,136 @@
|
|
1
|
+
|
2
|
+
# We've got to mock a beanstalkd server.
|
3
|
+
|
4
|
+
module Beanstalk
|
5
|
+
class TestJob
|
6
|
+
attr_reader :ttr, :body
|
7
|
+
|
8
|
+
def initialize(connection, name, body, ttr)
|
9
|
+
@connection = connection
|
10
|
+
@name = name
|
11
|
+
@body = body
|
12
|
+
@ttr = ttr
|
13
|
+
end
|
14
|
+
|
15
|
+
def delete
|
16
|
+
@connection.disturb!
|
17
|
+
|
18
|
+
@connection.queues[@name].shift
|
19
|
+
end
|
20
|
+
end
|
21
|
+
|
22
|
+
class Pool
|
23
|
+
@@connections = []
|
24
|
+
|
25
|
+
# Multiple pool servers are currently not implemented for testing purposes.
|
26
|
+
|
27
|
+
def self.new(server)
|
28
|
+
@connection = @@connections.detect { |connection| connection.server == server }
|
29
|
+
|
30
|
+
if @connection
|
31
|
+
@connection.up! if @connection.disconnected?
|
32
|
+
|
33
|
+
return @connection
|
34
|
+
end
|
35
|
+
|
36
|
+
unless @connection
|
37
|
+
@connection = TestConnection.new(server)
|
38
|
+
|
39
|
+
@@connections.push @connection
|
40
|
+
end
|
41
|
+
|
42
|
+
@connection
|
43
|
+
end
|
44
|
+
|
45
|
+
def self.clear!
|
46
|
+
@@connections = []
|
47
|
+
end
|
48
|
+
end
|
49
|
+
|
50
|
+
class TestConnection
|
51
|
+
attr_reader :server
|
52
|
+
|
53
|
+
def initialize(server)
|
54
|
+
@server = server
|
55
|
+
|
56
|
+
@availability = :available
|
57
|
+
@delay = 0
|
58
|
+
|
59
|
+
@watches = []
|
60
|
+
|
61
|
+
@queues = {}
|
62
|
+
end
|
63
|
+
|
64
|
+
def disturb!
|
65
|
+
raise Beanstalk::NotConnected unless available?
|
66
|
+
|
67
|
+
sleep @delay
|
68
|
+
end
|
69
|
+
|
70
|
+
def delay=(delay)
|
71
|
+
@delay = delay
|
72
|
+
end
|
73
|
+
|
74
|
+
def use(name)
|
75
|
+
disturb!
|
76
|
+
|
77
|
+
@current = name
|
78
|
+
end
|
79
|
+
|
80
|
+
def put(data, pri = 65536, delay = 0, ttr = 120)
|
81
|
+
disturb!
|
82
|
+
|
83
|
+
@queues[@current] ||= []
|
84
|
+
@queues[@current].push [data, pri, delay, ttr]
|
85
|
+
end
|
86
|
+
|
87
|
+
def watch(name)
|
88
|
+
disturb!
|
89
|
+
|
90
|
+
@watches.push name
|
91
|
+
end
|
92
|
+
|
93
|
+
def queues
|
94
|
+
@queues
|
95
|
+
end
|
96
|
+
|
97
|
+
def up!
|
98
|
+
@availability = :available
|
99
|
+
end
|
100
|
+
|
101
|
+
def available?
|
102
|
+
@availability == :available
|
103
|
+
end
|
104
|
+
|
105
|
+
def disconnect!
|
106
|
+
@availability = :disconnected
|
107
|
+
end
|
108
|
+
|
109
|
+
def disconnected?
|
110
|
+
@availability == :disconnected
|
111
|
+
end
|
112
|
+
|
113
|
+
def down!
|
114
|
+
@availability = :down
|
115
|
+
end
|
116
|
+
|
117
|
+
def down?
|
118
|
+
@availability == :down
|
119
|
+
end
|
120
|
+
|
121
|
+
def reserve
|
122
|
+
disturb!
|
123
|
+
|
124
|
+
@watches.each do |name|
|
125
|
+
if @queues[name] && !@queues[name].empty?
|
126
|
+
data, pri, delay, ttr = @queues[name].first
|
127
|
+
|
128
|
+
return TestJob.new(self, name, data, ttr)
|
129
|
+
end
|
130
|
+
end
|
131
|
+
|
132
|
+
nil
|
133
|
+
end
|
134
|
+
end
|
135
|
+
end
|
136
|
+
|
@@ -0,0 +1,208 @@
|
|
1
|
+
|
2
|
+
$:.unshift File.expand_path("../../lib", __FILE__)
|
3
|
+
|
4
|
+
require "stalking"
|
5
|
+
require "test/unit"
|
6
|
+
|
7
|
+
require File.expand_path("../beanstalk", __FILE__)
|
8
|
+
require File.expand_path("../test_logger", __FILE__)
|
9
|
+
|
10
|
+
class Stalking::ConsumerTest < Test::Unit::TestCase
|
11
|
+
def setup
|
12
|
+
Beanstalk::Pool.clear!
|
13
|
+
end
|
14
|
+
|
15
|
+
def test_before
|
16
|
+
Stalking::Producer.new.enqueue "test", "a" => "b"
|
17
|
+
|
18
|
+
res = nil
|
19
|
+
|
20
|
+
exception = assert_raises(Exception) do
|
21
|
+
Stalking::Consumer.new do
|
22
|
+
before do |name, args|
|
23
|
+
res = [name, args]
|
24
|
+
|
25
|
+
raise Exception
|
26
|
+
end
|
27
|
+
|
28
|
+
job "test" do |args|
|
29
|
+
# Nothing
|
30
|
+
end
|
31
|
+
end
|
32
|
+
end
|
33
|
+
|
34
|
+
assert_equal ["test", { "a" => "b" }], res
|
35
|
+
end
|
36
|
+
|
37
|
+
def test_after
|
38
|
+
Stalking::Producer.new.enqueue "test", "a" => "b"
|
39
|
+
|
40
|
+
res = nil
|
41
|
+
|
42
|
+
exception = assert_raises(Exception) do
|
43
|
+
Stalking::Consumer.new do
|
44
|
+
after do |name, args|
|
45
|
+
res = [name, args]
|
46
|
+
|
47
|
+
raise Exception
|
48
|
+
end
|
49
|
+
|
50
|
+
job "test" do |args|
|
51
|
+
# Nothing
|
52
|
+
end
|
53
|
+
end
|
54
|
+
end
|
55
|
+
|
56
|
+
assert_equal ["test", { "a" => "b" }], res
|
57
|
+
end
|
58
|
+
|
59
|
+
def test_error
|
60
|
+
Stalking::Producer.new.enqueue "test", "a" => "b"
|
61
|
+
|
62
|
+
res = nil
|
63
|
+
|
64
|
+
exception = assert_raises(Exception) do
|
65
|
+
Stalking::Consumer.new do
|
66
|
+
error do |e, name, args|
|
67
|
+
res = [e.message, name, args]
|
68
|
+
|
69
|
+
raise Exception
|
70
|
+
end
|
71
|
+
|
72
|
+
job "test" do |args|
|
73
|
+
raise "An error has occurred"
|
74
|
+
end
|
75
|
+
end
|
76
|
+
end
|
77
|
+
|
78
|
+
assert_equal ["An error has occurred", "test", { "a" => "b" }], res
|
79
|
+
end
|
80
|
+
|
81
|
+
def test_job
|
82
|
+
Stalking::Producer.new.enqueue "test", "a" => "b"
|
83
|
+
|
84
|
+
res = nil
|
85
|
+
|
86
|
+
exception = assert_raises(Exception) do
|
87
|
+
Stalking::Consumer.new do
|
88
|
+
job "test" do |args|
|
89
|
+
res = ["test", args]
|
90
|
+
|
91
|
+
raise Exception
|
92
|
+
end
|
93
|
+
end
|
94
|
+
end
|
95
|
+
|
96
|
+
assert_equal ["test", { "a" => "b" }], res
|
97
|
+
end
|
98
|
+
|
99
|
+
def test_work
|
100
|
+
Stalking::Producer.new.enqueue "test1", "a" => "b"
|
101
|
+
Stalking::Producer.new.enqueue "test2", "a" => "b"
|
102
|
+
Stalking::Producer.new.enqueue "test3", "a" => "b"
|
103
|
+
|
104
|
+
res = []
|
105
|
+
|
106
|
+
exception = assert_raises(Exception) do
|
107
|
+
Stalking::Consumer.new do
|
108
|
+
job "test1" do |args|
|
109
|
+
res.push "test1"
|
110
|
+
end
|
111
|
+
|
112
|
+
job "test2" do |args|
|
113
|
+
res.push "test2"
|
114
|
+
end
|
115
|
+
|
116
|
+
job "test3" do |args|
|
117
|
+
res.push "test3"
|
118
|
+
|
119
|
+
raise Exception
|
120
|
+
end
|
121
|
+
end
|
122
|
+
end
|
123
|
+
|
124
|
+
assert_equal ["test1", "test2", "test3"], res
|
125
|
+
end
|
126
|
+
|
127
|
+
def test_quit
|
128
|
+
raise NotImplementedError
|
129
|
+
end
|
130
|
+
|
131
|
+
def test_job_not_deleted
|
132
|
+
Stalking::Producer.new.enqueue "test", "a" => "b"
|
133
|
+
|
134
|
+
exception = assert_raises(Exception) do
|
135
|
+
Stalking::Consumer.new do
|
136
|
+
job "test" do |args|
|
137
|
+
raise Exception
|
138
|
+
end
|
139
|
+
end
|
140
|
+
end
|
141
|
+
|
142
|
+
res = nil
|
143
|
+
|
144
|
+
exception = assert_raises(Exception) do
|
145
|
+
Stalking::Consumer.new do
|
146
|
+
job "test" do |args|
|
147
|
+
res = ["test", args]
|
148
|
+
|
149
|
+
raise Exception
|
150
|
+
end
|
151
|
+
end
|
152
|
+
end
|
153
|
+
|
154
|
+
assert_equal ["test", {"a" => "b" }], res
|
155
|
+
end
|
156
|
+
|
157
|
+
def test_job_deleted
|
158
|
+
Stalking::Producer.new.enqueue "test", "a" => "b"
|
159
|
+
|
160
|
+
res = nil
|
161
|
+
|
162
|
+
exception = assert_raises(NoMethodError) do
|
163
|
+
Stalking::Consumer.new do
|
164
|
+
job "test" do |args|
|
165
|
+
res = ["test", args]
|
166
|
+
end
|
167
|
+
end
|
168
|
+
end
|
169
|
+
|
170
|
+
assert_equal ["test", { "a" => "b" }], res
|
171
|
+
assert_equal "undefined method `body' for nil:NilClass", exception.message
|
172
|
+
end
|
173
|
+
|
174
|
+
def test_reconnect
|
175
|
+
Stalking::Producer.new.enqueue "test", "a" => "b"
|
176
|
+
|
177
|
+
logger = TestLogger.new
|
178
|
+
|
179
|
+
res = []
|
180
|
+
|
181
|
+
exception = assert_raises(Exception) do
|
182
|
+
Stalking::Consumer.new(:logger => logger) do
|
183
|
+
job "test" do |args|
|
184
|
+
if res.empty?
|
185
|
+
res.push ["down"]
|
186
|
+
|
187
|
+
# Simulate a temporary beanstalk connection error.
|
188
|
+
|
189
|
+
Beanstalk::Pool.new(["localhost:11300"]).disconnect!
|
190
|
+
else
|
191
|
+
res.push ["test", args]
|
192
|
+
|
193
|
+
raise Exception
|
194
|
+
end
|
195
|
+
end
|
196
|
+
end
|
197
|
+
end
|
198
|
+
|
199
|
+
assert_equal [["down"], ["test", { "a" => "b" }]], res
|
200
|
+
|
201
|
+
assert_equal ["Stalking error: Beanstalk::NotConnected"], logger.messages
|
202
|
+
end
|
203
|
+
|
204
|
+
def test_logger
|
205
|
+
# Already tested
|
206
|
+
end
|
207
|
+
end
|
208
|
+
|
@@ -0,0 +1,86 @@
|
|
1
|
+
|
2
|
+
$:.unshift File.expand_path("../../lib", __FILE__)
|
3
|
+
|
4
|
+
require "stalking"
|
5
|
+
require "test/unit"
|
6
|
+
|
7
|
+
require File.expand_path("../test_logger", __FILE__)
|
8
|
+
|
9
|
+
class Stalking::ProducerTest < Test::Unit::TestCase
|
10
|
+
def setup
|
11
|
+
Beanstalk::Pool.clear!
|
12
|
+
end
|
13
|
+
|
14
|
+
def test_enqueue
|
15
|
+
Stalking::Producer.new.enqueue "test", { "key" => "value" }, :pri => 100, :delay => 100, :ttr => 100
|
16
|
+
|
17
|
+
assert_equal ['["test",{"key":"value"}]', 100, 100, 100], Beanstalk::Pool.new(["localhost:11300"]).queues["test"].last
|
18
|
+
end
|
19
|
+
|
20
|
+
def test_defaults
|
21
|
+
Stalking::Producer.new.enqueue "test", "key" => "value"
|
22
|
+
|
23
|
+
assert_equal ['["test",{"key":"value"}]', 65536, 0, 120], Beanstalk::Pool.new(["localhost:11300"]).queues["test"].last
|
24
|
+
end
|
25
|
+
|
26
|
+
def test_reconnect
|
27
|
+
producer = Stalking::Producer.new(:servers => ["server1", "server2"])
|
28
|
+
|
29
|
+
server1 = Beanstalk::Pool.new(["server1"])
|
30
|
+
server2 = Beanstalk::Pool.new(["server2"])
|
31
|
+
|
32
|
+
srand 0
|
33
|
+
|
34
|
+
producer.enqueue "test", "up" => "true"
|
35
|
+
|
36
|
+
assert_equal ['["test",{"up":"true"}]', 65536, 0, 120], server1.queues["test"].last
|
37
|
+
|
38
|
+
srand 0
|
39
|
+
|
40
|
+
server1.disconnect!
|
41
|
+
|
42
|
+
producer.enqueue "test", "down" => "disconnected"
|
43
|
+
|
44
|
+
assert_equal ['["test",{"down":"disconnected"}]', 65536, 0, 120], server1.queues["test"].last
|
45
|
+
|
46
|
+
server1.down!
|
47
|
+
|
48
|
+
producer.enqueue "test", "down" => "permanent"
|
49
|
+
|
50
|
+
assert_equal ['["test",{"down":"permanent"}]', 65536, 0, 120], server2.queues["test"].last
|
51
|
+
end
|
52
|
+
|
53
|
+
def test_timeout
|
54
|
+
producer = Stalking::Producer.new(:servers => ["server1", "server2"])
|
55
|
+
|
56
|
+
server1 = Beanstalk::Pool.new(["server1"])
|
57
|
+
server2 = Beanstalk::Pool.new(["server2"])
|
58
|
+
|
59
|
+
srand 0
|
60
|
+
|
61
|
+
producer.enqueue "test", "up" => "true"
|
62
|
+
|
63
|
+
assert_equal ['["test",{"up":"true"}]', 65536, 0, 120], server1.queues["test"].last
|
64
|
+
|
65
|
+
srand 0
|
66
|
+
|
67
|
+
server1.delay = 5
|
68
|
+
|
69
|
+
producer.enqueue "test", "down" => "delayed"
|
70
|
+
|
71
|
+
assert_equal ['["test",{"down":"delayed"}]', 65536, 0, 120], server2.queues["test"].last
|
72
|
+
end
|
73
|
+
|
74
|
+
def test_logger
|
75
|
+
logger = TestLogger.new
|
76
|
+
|
77
|
+
producer = Stalking::Producer.new(:logger => logger)
|
78
|
+
|
79
|
+
Beanstalk::Pool.new(["localhost:11300"]).down!
|
80
|
+
|
81
|
+
producer.enqueue "test", "up" => "true"
|
82
|
+
|
83
|
+
assert_equal ["Stalking error: Beanstalk::NotConnected"], logger.messages
|
84
|
+
end
|
85
|
+
end
|
86
|
+
|
metadata
ADDED
@@ -0,0 +1,111 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: stalking
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.0.1
|
5
|
+
prerelease:
|
6
|
+
platform: ruby
|
7
|
+
authors:
|
8
|
+
- Benjamin Vetter
|
9
|
+
autorequire:
|
10
|
+
bindir: bin
|
11
|
+
cert_chain: []
|
12
|
+
date: 2013-06-28 00:00:00.000000000 Z
|
13
|
+
dependencies:
|
14
|
+
- !ruby/object:Gem::Dependency
|
15
|
+
name: rake
|
16
|
+
requirement: !ruby/object:Gem::Requirement
|
17
|
+
none: false
|
18
|
+
requirements:
|
19
|
+
- - ! '>='
|
20
|
+
- !ruby/object:Gem::Version
|
21
|
+
version: '0'
|
22
|
+
type: :development
|
23
|
+
prerelease: false
|
24
|
+
version_requirements: !ruby/object:Gem::Requirement
|
25
|
+
none: false
|
26
|
+
requirements:
|
27
|
+
- - ! '>='
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
version: '0'
|
30
|
+
- !ruby/object:Gem::Dependency
|
31
|
+
name: beanstalk-client
|
32
|
+
requirement: !ruby/object:Gem::Requirement
|
33
|
+
none: false
|
34
|
+
requirements:
|
35
|
+
- - ! '>='
|
36
|
+
- !ruby/object:Gem::Version
|
37
|
+
version: '0'
|
38
|
+
type: :runtime
|
39
|
+
prerelease: false
|
40
|
+
version_requirements: !ruby/object:Gem::Requirement
|
41
|
+
none: false
|
42
|
+
requirements:
|
43
|
+
- - ! '>='
|
44
|
+
- !ruby/object:Gem::Version
|
45
|
+
version: '0'
|
46
|
+
- !ruby/object:Gem::Dependency
|
47
|
+
name: json_pure
|
48
|
+
requirement: !ruby/object:Gem::Requirement
|
49
|
+
none: false
|
50
|
+
requirements:
|
51
|
+
- - ! '>='
|
52
|
+
- !ruby/object:Gem::Version
|
53
|
+
version: '0'
|
54
|
+
type: :runtime
|
55
|
+
prerelease: false
|
56
|
+
version_requirements: !ruby/object:Gem::Requirement
|
57
|
+
none: false
|
58
|
+
requirements:
|
59
|
+
- - ! '>='
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '0'
|
62
|
+
description: Asynchronous job processing using the beanstalkd queue server with automatic
|
63
|
+
reconnects and failover
|
64
|
+
email:
|
65
|
+
- vetter@flakks.com
|
66
|
+
executables:
|
67
|
+
- stalking
|
68
|
+
extensions: []
|
69
|
+
extra_rdoc_files: []
|
70
|
+
files:
|
71
|
+
- .gitignore
|
72
|
+
- Gemfile
|
73
|
+
- LICENSE.txt
|
74
|
+
- README.md
|
75
|
+
- Rakefile
|
76
|
+
- bin/stalking
|
77
|
+
- lib/stalking.rb
|
78
|
+
- lib/stalking/consumer.rb
|
79
|
+
- lib/stalking/handlers.rb
|
80
|
+
- lib/stalking/producer.rb
|
81
|
+
- lib/stalking/version.rb
|
82
|
+
- stalking.gemspec
|
83
|
+
- test/stalking/beanstalk.rb
|
84
|
+
- test/stalking/consumer_test.rb
|
85
|
+
- test/stalking/producer_test.rb
|
86
|
+
- test/stalking/test_logger.rb
|
87
|
+
homepage: https://github.com/mrkamel/stalking
|
88
|
+
licenses: []
|
89
|
+
post_install_message:
|
90
|
+
rdoc_options: []
|
91
|
+
require_paths:
|
92
|
+
- lib
|
93
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
94
|
+
none: false
|
95
|
+
requirements:
|
96
|
+
- - ! '>='
|
97
|
+
- !ruby/object:Gem::Version
|
98
|
+
version: '0'
|
99
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
100
|
+
none: false
|
101
|
+
requirements:
|
102
|
+
- - ! '>='
|
103
|
+
- !ruby/object:Gem::Version
|
104
|
+
version: '0'
|
105
|
+
requirements: []
|
106
|
+
rubyforge_project:
|
107
|
+
rubygems_version: 1.8.23
|
108
|
+
signing_key:
|
109
|
+
specification_version: 3
|
110
|
+
summary: Asynchronous jobs using beanstalkd with automatic reconnects and failover
|
111
|
+
test_files: []
|