stalking 0.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/.gitignore +17 -0
- data/Gemfile +4 -0
- data/LICENSE.txt +22 -0
- data/README.md +187 -0
- data/Rakefile +9 -0
- data/bin/stalking +11 -0
- data/lib/stalking.rb +10 -0
- data/lib/stalking/consumer.rb +117 -0
- data/lib/stalking/handlers.rb +31 -0
- data/lib/stalking/producer.rb +74 -0
- data/lib/stalking/version.rb +3 -0
- data/stalking.gemspec +24 -0
- data/test/stalking/beanstalk.rb +136 -0
- data/test/stalking/consumer_test.rb +208 -0
- data/test/stalking/producer_test.rb +86 -0
- data/test/stalking/test_logger.rb +13 -0
- metadata +111 -0
data/.gitignore
ADDED
data/Gemfile
ADDED
data/LICENSE.txt
ADDED
@@ -0,0 +1,22 @@
|
|
1
|
+
Copyright (c) 2012 Benjamin Vetter
|
2
|
+
|
3
|
+
MIT License
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
6
|
+
a copy of this software and associated documentation files (the
|
7
|
+
"Software"), to deal in the Software without restriction, including
|
8
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
9
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
10
|
+
permit persons to whom the Software is furnished to do so, subject to
|
11
|
+
the following conditions:
|
12
|
+
|
13
|
+
The above copyright notice and this permission notice shall be
|
14
|
+
included in all copies or substantial portions of the Software.
|
15
|
+
|
16
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
17
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
18
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
19
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
20
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
21
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
22
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,187 @@
|
|
1
|
+
|
2
|
+
# Stalking
|
3
|
+
|
4
|
+
Asynchronous job processing using the beanstalkd queue server with automatic
|
5
|
+
reconnects and failover for your ruby projects.
|
6
|
+
|
7
|
+
Background jobs are essential to every project these days and the tools should
|
8
|
+
be as reliable as possible.
|
9
|
+
|
10
|
+
Beanstalkd is perfectly suited for background processing and the stalker gem
|
11
|
+
has long been my favorite to integrate beanstalkd into my projects.
|
12
|
+
Unfortunately, stalker does not provide automatic reconnects and failover.
|
13
|
+
|
14
|
+
The DSL provided by stalking is very similar to the DSL of stalker with only
|
15
|
+
minimal changes driven by design decisions and its additional feature set.
|
16
|
+
|
17
|
+
## Installation
|
18
|
+
|
19
|
+
Add this line to your application's Gemfile:
|
20
|
+
|
21
|
+
gem 'stalking', :git => 'git://github.com/mrkamel/stalking.git'
|
22
|
+
|
23
|
+
And then execute:
|
24
|
+
|
25
|
+
$ bundle
|
26
|
+
|
27
|
+
## Usage
|
28
|
+
|
29
|
+
Stalking provides two modes of operation, a producer and a consumer mode.
|
30
|
+
|
31
|
+
Enqueuing new jobs is as easy as:
|
32
|
+
|
33
|
+
```ruby
|
34
|
+
Stalking::Producer.new.enqueue "name", "key" => "value"
|
35
|
+
```
|
36
|
+
|
37
|
+
This will try to connect to a beanstalkd instance running on `localhost:11300`
|
38
|
+
to enqueue a new job named `job` with the provided arguments. The arguments
|
39
|
+
will be serialized using JSON before being enqueued.
|
40
|
+
|
41
|
+
To specify a `priority`, `ttr` or `delay`, please use:
|
42
|
+
|
43
|
+
```ruby
|
44
|
+
Stalking::Producer.new.enqueue "name", { "key" => "value" }, :ttr => 360, :delay => 60, :pri => 100
|
45
|
+
```
|
46
|
+
|
47
|
+
You can of course use multiple beanstalkd instances, by using:
|
48
|
+
|
49
|
+
```ruby
|
50
|
+
producer = Stalking::Producer.new(:servers => ["192.168.1.100:11300", "192.168.1.101"])
|
51
|
+
producer.enqueue "name", "key" => "value"
|
52
|
+
```
|
53
|
+
|
54
|
+
Here, everytime you enqueue a new job, stalking will randomly choose a server
|
55
|
+
from your list to send the job to. Thus, you can e.g. create a producer for
|
56
|
+
local jobs (like e.g. for local background image processing) and another
|
57
|
+
producer for global jobs:
|
58
|
+
|
59
|
+
```ruby
|
60
|
+
global_queue = Stalking::Producer.new(:servers => ["192.168.1.100:11300", "192.168.1.101"])
|
61
|
+
local_queue = Stalking::Producer.new
|
62
|
+
```
|
63
|
+
|
64
|
+
If a server from your list is down or unreachable, stalking will use another
|
65
|
+
one from your list. Only in case all your servers are down, `enqueue` will
|
66
|
+
return false. However, please carefully read the `Caveats` section.
|
67
|
+
|
68
|
+
You can even set the `ttr`, `priority` and `delay` options
|
69
|
+
on a per-producer base:
|
70
|
+
|
71
|
+
```ruby
|
72
|
+
producer = Stalking::Producer.new(:pri => 100, :delay => 60, :ttr => 360)
|
73
|
+
```
|
74
|
+
|
75
|
+
To set an upper limit for servers to be tried out for queuing use the `tries`
|
76
|
+
option:
|
77
|
+
|
78
|
+
```ruby
|
79
|
+
producer = Stalking::Producer.new(:servers => [..., ..., ...], :tries => 2)
|
80
|
+
```
|
81
|
+
|
82
|
+
To consume jobs in a background worker, please use the following DSL, which is
|
83
|
+
quite similar to stalker's DSL:
|
84
|
+
|
85
|
+
```ruby
|
86
|
+
Stalking::Consumer.new do
|
87
|
+
job "name" do |args|
|
88
|
+
# Do something
|
89
|
+
end
|
90
|
+
|
91
|
+
job "another" do |args|
|
92
|
+
# Do something
|
93
|
+
end
|
94
|
+
|
95
|
+
...
|
96
|
+
end
|
97
|
+
```
|
98
|
+
|
99
|
+
After being specified, the consumer will immediately block and wait to consume
|
100
|
+
jobs from beanstalkd.
|
101
|
+
|
102
|
+
Additionally, you can specify before, after and error hooks:
|
103
|
+
|
104
|
+
```ruby
|
105
|
+
Stalking::Consumer.new do
|
106
|
+
before do |name, args|
|
107
|
+
# Executed before a job runs
|
108
|
+
end
|
109
|
+
|
110
|
+
after do |name, args|
|
111
|
+
# Executed after a job has run
|
112
|
+
end
|
113
|
+
|
114
|
+
error do |e, name, args|
|
115
|
+
# Executed when job exited abnormally
|
116
|
+
end
|
117
|
+
|
118
|
+
job "name" do |args|
|
119
|
+
# The job itself
|
120
|
+
end
|
121
|
+
end
|
122
|
+
```
|
123
|
+
|
124
|
+
If you save this file to e.g. `jobs.rb`, you can execute your worker like:
|
125
|
+
|
126
|
+
```
|
127
|
+
$ stalking jobs
|
128
|
+
```
|
129
|
+
|
130
|
+
Again, by default stalking connects to localhost:11300, but you can specify a
|
131
|
+
pool of servers to consume jobs from:
|
132
|
+
|
133
|
+
```ruby
|
134
|
+
Stalking::Consumer.new(:servers => ["192.168.1.100:11300", "192.168.1.101:11300"]) do
|
135
|
+
...
|
136
|
+
end
|
137
|
+
```
|
138
|
+
|
139
|
+
Please note that, due to the implementation of beanstalk-client, the consumer
|
140
|
+
requires all pool servers to be up. Otherwise, the consumer will enter a
|
141
|
+
reconnect loop, until all servers are up again. Thus, stalking will try to
|
142
|
+
reconnect to the servers when a connection is lost and it's recommended for
|
143
|
+
consumers to connect to a single server, usually.
|
144
|
+
|
145
|
+
That should be it.
|
146
|
+
|
147
|
+
## Caveats
|
148
|
+
|
149
|
+
Having producers which try to reconnect and/or try to connect to other servers
|
150
|
+
when a single server is down, of course require some time for the error
|
151
|
+
detection and processing. This usually should not be a big issue, except there
|
152
|
+
is some temporary lag between the producer and the beanstalkd server. The
|
153
|
+
producer then has to wait for a timeout and while the producer is waiting for
|
154
|
+
the timeout it will block your application. The default timeout is 250
|
155
|
+
milliseconds, but you can change the timeout via:
|
156
|
+
|
157
|
+
```ruby
|
158
|
+
Stalking::Producer.new :timeout => 0.1 # Set timeout to 100 milliseconds
|
159
|
+
```
|
160
|
+
|
161
|
+
The optimal timeout for your setup depends on your network connection, the
|
162
|
+
distance of your servers, etc. You should experiment a bit with it if you
|
163
|
+
experience some kind of problems. However, i don't emphasize to use stalking
|
164
|
+
producers which communicate with beanstalkd servers over WAN connections.
|
165
|
+
Only local, data-center connections to beanstalkd servers should be used.
|
166
|
+
|
167
|
+
# Error Logging
|
168
|
+
|
169
|
+
To log connection errors, you can provide a logger to the consumers as well
|
170
|
+
as producers:
|
171
|
+
|
172
|
+
```ruby
|
173
|
+
Stalking::Producer.new :logger => ...
|
174
|
+
Stalking::Consumer.new :logger => ...
|
175
|
+
```
|
176
|
+
|
177
|
+
You can hand out any kind of logging object, which responds to an `error`
|
178
|
+
method and takes a string error message.
|
179
|
+
|
180
|
+
## Contributing
|
181
|
+
|
182
|
+
1. Fork it
|
183
|
+
2. Create your feature branch (`git checkout -b my-new-feature`)
|
184
|
+
3. Commit your changes (`git commit -am 'Add some feature'`)
|
185
|
+
4. Push to the branch (`git push origin my-new-feature`)
|
186
|
+
5. Create new Pull Request
|
187
|
+
|
data/Rakefile
ADDED
data/bin/stalking
ADDED
data/lib/stalking.rb
ADDED
@@ -0,0 +1,117 @@
|
|
1
|
+
|
2
|
+
module Stalking
|
3
|
+
class Consumer
|
4
|
+
def initialize(options = {}, &block)
|
5
|
+
@servers = options[:servers] || ["localhost:11300"]
|
6
|
+
|
7
|
+
@logger = options[:logger]
|
8
|
+
|
9
|
+
@handlers = Handlers.new
|
10
|
+
|
11
|
+
if block_given?
|
12
|
+
@handlers.instance_eval &block
|
13
|
+
|
14
|
+
work
|
15
|
+
end
|
16
|
+
end
|
17
|
+
|
18
|
+
def handle(handlers, name, args)
|
19
|
+
handlers.each do |handler|
|
20
|
+
handler.call name, args
|
21
|
+
end
|
22
|
+
end
|
23
|
+
|
24
|
+
def handle_error(e, name, args)
|
25
|
+
@handlers.error_handlers.each do |handler|
|
26
|
+
handler.call e, name, args
|
27
|
+
end
|
28
|
+
end
|
29
|
+
|
30
|
+
private
|
31
|
+
|
32
|
+
def work
|
33
|
+
setup
|
34
|
+
|
35
|
+
begin
|
36
|
+
loop do
|
37
|
+
job = connection.reserve
|
38
|
+
|
39
|
+
exit if @quit
|
40
|
+
|
41
|
+
busy do
|
42
|
+
name, args = JSON.parse(job.body)
|
43
|
+
|
44
|
+
if handler = @handlers.job_handlers[name]
|
45
|
+
begin
|
46
|
+
Timeout::timeout(job.ttr - 1) do
|
47
|
+
handle @handlers.before_handlers, name, args
|
48
|
+
|
49
|
+
handler.call args
|
50
|
+
|
51
|
+
handle @handlers.after_handlers, name, args
|
52
|
+
end
|
53
|
+
rescue Beanstalk::NotConnected => e
|
54
|
+
raise e # Re-raise
|
55
|
+
rescue Timeout::Error, StandardError => e
|
56
|
+
handle_error e, name, args
|
57
|
+
end
|
58
|
+
end
|
59
|
+
end
|
60
|
+
|
61
|
+
job.delete
|
62
|
+
end
|
63
|
+
rescue Beanstalk::NotConnected => e
|
64
|
+
log_error e
|
65
|
+
|
66
|
+
sleep 1
|
67
|
+
|
68
|
+
reconnect
|
69
|
+
|
70
|
+
retry
|
71
|
+
end
|
72
|
+
end
|
73
|
+
|
74
|
+
def setup
|
75
|
+
trap "QUIT" do
|
76
|
+
exit unless @busy
|
77
|
+
|
78
|
+
@quit = true
|
79
|
+
end
|
80
|
+
end
|
81
|
+
|
82
|
+
def busy
|
83
|
+
@busy = true
|
84
|
+
|
85
|
+
yield
|
86
|
+
ensure
|
87
|
+
@busy = false
|
88
|
+
end
|
89
|
+
|
90
|
+
def connect
|
91
|
+
@connection ||= Beanstalk::Pool.new(@servers)
|
92
|
+
|
93
|
+
@handlers.job_handlers.keys.each { |name| @connection.watch name }
|
94
|
+
|
95
|
+
@connection
|
96
|
+
rescue Beanstalk::NotConnected => e
|
97
|
+
log_error e
|
98
|
+
|
99
|
+
sleep 1
|
100
|
+
|
101
|
+
retry
|
102
|
+
end
|
103
|
+
|
104
|
+
alias :connection :connect
|
105
|
+
|
106
|
+
def reconnect
|
107
|
+
@connection = nil
|
108
|
+
|
109
|
+
connect
|
110
|
+
end
|
111
|
+
|
112
|
+
def log_error(e)
|
113
|
+
@logger.error("Stalking error: #{e.message}") if @logger
|
114
|
+
end
|
115
|
+
end
|
116
|
+
end
|
117
|
+
|
@@ -0,0 +1,31 @@
|
|
1
|
+
|
2
|
+
module Stalking
|
3
|
+
class Handlers
|
4
|
+
attr_reader :before_handlers, :after_handlers, :error_handlers, :job_handlers
|
5
|
+
|
6
|
+
def initialize
|
7
|
+
@before_handlers = []
|
8
|
+
@after_handlers = []
|
9
|
+
@error_handlers = []
|
10
|
+
|
11
|
+
@job_handlers = {}
|
12
|
+
end
|
13
|
+
|
14
|
+
def before(&block)
|
15
|
+
@before_handlers.push block
|
16
|
+
end
|
17
|
+
|
18
|
+
def after(&block)
|
19
|
+
@after_handlers.push block
|
20
|
+
end
|
21
|
+
|
22
|
+
def error(&block)
|
23
|
+
@error_handlers.push block
|
24
|
+
end
|
25
|
+
|
26
|
+
def job(name, &block)
|
27
|
+
@job_handlers[name] = block
|
28
|
+
end
|
29
|
+
end
|
30
|
+
end
|
31
|
+
|
@@ -0,0 +1,74 @@
|
|
1
|
+
|
2
|
+
module Stalking
|
3
|
+
class Producer
|
4
|
+
def initialize(options = {}, &block)
|
5
|
+
@timeout = options[:timeout] || 0.25
|
6
|
+
|
7
|
+
@pri = options[:pri] || 65536
|
8
|
+
@delay = options[:delay] || 0
|
9
|
+
@ttr = options[:ttr] || 120
|
10
|
+
|
11
|
+
@logger = options[:logger]
|
12
|
+
|
13
|
+
@servers = options[:servers] || ["localhost:11300"]
|
14
|
+
|
15
|
+
@tries = options[:tries] || @servers.size
|
16
|
+
|
17
|
+
@connections = {}
|
18
|
+
end
|
19
|
+
|
20
|
+
def enqueue(name, args = {}, options = {})
|
21
|
+
# Send the job to a random server.
|
22
|
+
|
23
|
+
@servers.shuffle.first(@tries).each do |server|
|
24
|
+
begin
|
25
|
+
Timeout::timeout(@timeout) do
|
26
|
+
@connections[server] ||= Beanstalk::Pool.new([server])
|
27
|
+
|
28
|
+
enq @connections[server], name, args, options
|
29
|
+
end
|
30
|
+
|
31
|
+
return true
|
32
|
+
rescue Beanstalk::NotConnected
|
33
|
+
# Connecting to the beanstalk server has failed.
|
34
|
+
# Let's try to reconnect and enqueue afterwards.
|
35
|
+
|
36
|
+
begin
|
37
|
+
Timeout::timeout(@timeout) do
|
38
|
+
@connections[server] = Beanstalk::Pool.new([server])
|
39
|
+
|
40
|
+
enq @connections[server], name, args, options
|
41
|
+
end
|
42
|
+
|
43
|
+
return true
|
44
|
+
rescue Beanstalk::NotConnected, Timeout::Error, StandardError => e
|
45
|
+
log_error e
|
46
|
+
end
|
47
|
+
rescue Timeout::Error, StandardError => e
|
48
|
+
log_error e
|
49
|
+
end
|
50
|
+
end
|
51
|
+
|
52
|
+
# All servers are currently unavailable.
|
53
|
+
# Enqueuing the job has failed.
|
54
|
+
|
55
|
+
false
|
56
|
+
end
|
57
|
+
|
58
|
+
private
|
59
|
+
|
60
|
+
def enq(connection, name, args = {}, options = {})
|
61
|
+
pri = options[:pri] || @pri
|
62
|
+
delay = options[:delay] || @delay
|
63
|
+
ttr = options[:ttr] || @ttr
|
64
|
+
|
65
|
+
connection.use name
|
66
|
+
connection.put [name, args].to_json, pri, delay, ttr
|
67
|
+
end
|
68
|
+
|
69
|
+
def log_error(e)
|
70
|
+
@logger.error("Stalking error: #{e.message}") if @logger
|
71
|
+
end
|
72
|
+
end
|
73
|
+
end
|
74
|
+
|
data/stalking.gemspec
ADDED
@@ -0,0 +1,24 @@
|
|
1
|
+
# -*- encoding: utf-8 -*-
|
2
|
+
$:.push File.expand_path("../lib", __FILE__)
|
3
|
+
require "stalking/version"
|
4
|
+
|
5
|
+
Gem::Specification.new do |s|
|
6
|
+
s.name = "stalking"
|
7
|
+
s.version = Stalking::VERSION
|
8
|
+
s.authors = ["Benjamin Vetter"]
|
9
|
+
s.email = ["vetter@flakks.com"]
|
10
|
+
s.homepage = "https://github.com/mrkamel/stalking"
|
11
|
+
s.summary = %q{Asynchronous jobs using beanstalkd with automatic reconnects and failover}
|
12
|
+
s.description = %q{Asynchronous job processing using the beanstalkd queue server with automatic reconnects and failover}
|
13
|
+
|
14
|
+
s.files = `git ls-files`.split("\n")
|
15
|
+
s.test_files = `git ls-files -- {test,spec,features}/*`.split("\n")
|
16
|
+
s.executables = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
|
17
|
+
s.require_paths = ["lib"]
|
18
|
+
|
19
|
+
s.add_development_dependency "rake"
|
20
|
+
|
21
|
+
s.add_dependency "beanstalk-client"
|
22
|
+
s.add_dependency "json_pure"
|
23
|
+
end
|
24
|
+
|
@@ -0,0 +1,136 @@
|
|
1
|
+
|
2
|
+
# We've got to mock a beanstalkd server.
|
3
|
+
|
4
|
+
module Beanstalk
|
5
|
+
class TestJob
|
6
|
+
attr_reader :ttr, :body
|
7
|
+
|
8
|
+
def initialize(connection, name, body, ttr)
|
9
|
+
@connection = connection
|
10
|
+
@name = name
|
11
|
+
@body = body
|
12
|
+
@ttr = ttr
|
13
|
+
end
|
14
|
+
|
15
|
+
def delete
|
16
|
+
@connection.disturb!
|
17
|
+
|
18
|
+
@connection.queues[@name].shift
|
19
|
+
end
|
20
|
+
end
|
21
|
+
|
22
|
+
class Pool
|
23
|
+
@@connections = []
|
24
|
+
|
25
|
+
# Multiple pool servers are currently not implemented for testing purposes.
|
26
|
+
|
27
|
+
def self.new(server)
|
28
|
+
@connection = @@connections.detect { |connection| connection.server == server }
|
29
|
+
|
30
|
+
if @connection
|
31
|
+
@connection.up! if @connection.disconnected?
|
32
|
+
|
33
|
+
return @connection
|
34
|
+
end
|
35
|
+
|
36
|
+
unless @connection
|
37
|
+
@connection = TestConnection.new(server)
|
38
|
+
|
39
|
+
@@connections.push @connection
|
40
|
+
end
|
41
|
+
|
42
|
+
@connection
|
43
|
+
end
|
44
|
+
|
45
|
+
def self.clear!
|
46
|
+
@@connections = []
|
47
|
+
end
|
48
|
+
end
|
49
|
+
|
50
|
+
class TestConnection
|
51
|
+
attr_reader :server
|
52
|
+
|
53
|
+
def initialize(server)
|
54
|
+
@server = server
|
55
|
+
|
56
|
+
@availability = :available
|
57
|
+
@delay = 0
|
58
|
+
|
59
|
+
@watches = []
|
60
|
+
|
61
|
+
@queues = {}
|
62
|
+
end
|
63
|
+
|
64
|
+
def disturb!
|
65
|
+
raise Beanstalk::NotConnected unless available?
|
66
|
+
|
67
|
+
sleep @delay
|
68
|
+
end
|
69
|
+
|
70
|
+
def delay=(delay)
|
71
|
+
@delay = delay
|
72
|
+
end
|
73
|
+
|
74
|
+
def use(name)
|
75
|
+
disturb!
|
76
|
+
|
77
|
+
@current = name
|
78
|
+
end
|
79
|
+
|
80
|
+
def put(data, pri = 65536, delay = 0, ttr = 120)
|
81
|
+
disturb!
|
82
|
+
|
83
|
+
@queues[@current] ||= []
|
84
|
+
@queues[@current].push [data, pri, delay, ttr]
|
85
|
+
end
|
86
|
+
|
87
|
+
def watch(name)
|
88
|
+
disturb!
|
89
|
+
|
90
|
+
@watches.push name
|
91
|
+
end
|
92
|
+
|
93
|
+
def queues
|
94
|
+
@queues
|
95
|
+
end
|
96
|
+
|
97
|
+
def up!
|
98
|
+
@availability = :available
|
99
|
+
end
|
100
|
+
|
101
|
+
def available?
|
102
|
+
@availability == :available
|
103
|
+
end
|
104
|
+
|
105
|
+
def disconnect!
|
106
|
+
@availability = :disconnected
|
107
|
+
end
|
108
|
+
|
109
|
+
def disconnected?
|
110
|
+
@availability == :disconnected
|
111
|
+
end
|
112
|
+
|
113
|
+
def down!
|
114
|
+
@availability = :down
|
115
|
+
end
|
116
|
+
|
117
|
+
def down?
|
118
|
+
@availability == :down
|
119
|
+
end
|
120
|
+
|
121
|
+
def reserve
|
122
|
+
disturb!
|
123
|
+
|
124
|
+
@watches.each do |name|
|
125
|
+
if @queues[name] && !@queues[name].empty?
|
126
|
+
data, pri, delay, ttr = @queues[name].first
|
127
|
+
|
128
|
+
return TestJob.new(self, name, data, ttr)
|
129
|
+
end
|
130
|
+
end
|
131
|
+
|
132
|
+
nil
|
133
|
+
end
|
134
|
+
end
|
135
|
+
end
|
136
|
+
|
@@ -0,0 +1,208 @@
|
|
1
|
+
|
2
|
+
$:.unshift File.expand_path("../../lib", __FILE__)
|
3
|
+
|
4
|
+
require "stalking"
|
5
|
+
require "test/unit"
|
6
|
+
|
7
|
+
require File.expand_path("../beanstalk", __FILE__)
|
8
|
+
require File.expand_path("../test_logger", __FILE__)
|
9
|
+
|
10
|
+
class Stalking::ConsumerTest < Test::Unit::TestCase
|
11
|
+
def setup
|
12
|
+
Beanstalk::Pool.clear!
|
13
|
+
end
|
14
|
+
|
15
|
+
def test_before
|
16
|
+
Stalking::Producer.new.enqueue "test", "a" => "b"
|
17
|
+
|
18
|
+
res = nil
|
19
|
+
|
20
|
+
exception = assert_raises(Exception) do
|
21
|
+
Stalking::Consumer.new do
|
22
|
+
before do |name, args|
|
23
|
+
res = [name, args]
|
24
|
+
|
25
|
+
raise Exception
|
26
|
+
end
|
27
|
+
|
28
|
+
job "test" do |args|
|
29
|
+
# Nothing
|
30
|
+
end
|
31
|
+
end
|
32
|
+
end
|
33
|
+
|
34
|
+
assert_equal ["test", { "a" => "b" }], res
|
35
|
+
end
|
36
|
+
|
37
|
+
def test_after
|
38
|
+
Stalking::Producer.new.enqueue "test", "a" => "b"
|
39
|
+
|
40
|
+
res = nil
|
41
|
+
|
42
|
+
exception = assert_raises(Exception) do
|
43
|
+
Stalking::Consumer.new do
|
44
|
+
after do |name, args|
|
45
|
+
res = [name, args]
|
46
|
+
|
47
|
+
raise Exception
|
48
|
+
end
|
49
|
+
|
50
|
+
job "test" do |args|
|
51
|
+
# Nothing
|
52
|
+
end
|
53
|
+
end
|
54
|
+
end
|
55
|
+
|
56
|
+
assert_equal ["test", { "a" => "b" }], res
|
57
|
+
end
|
58
|
+
|
59
|
+
def test_error
|
60
|
+
Stalking::Producer.new.enqueue "test", "a" => "b"
|
61
|
+
|
62
|
+
res = nil
|
63
|
+
|
64
|
+
exception = assert_raises(Exception) do
|
65
|
+
Stalking::Consumer.new do
|
66
|
+
error do |e, name, args|
|
67
|
+
res = [e.message, name, args]
|
68
|
+
|
69
|
+
raise Exception
|
70
|
+
end
|
71
|
+
|
72
|
+
job "test" do |args|
|
73
|
+
raise "An error has occurred"
|
74
|
+
end
|
75
|
+
end
|
76
|
+
end
|
77
|
+
|
78
|
+
assert_equal ["An error has occurred", "test", { "a" => "b" }], res
|
79
|
+
end
|
80
|
+
|
81
|
+
def test_job
|
82
|
+
Stalking::Producer.new.enqueue "test", "a" => "b"
|
83
|
+
|
84
|
+
res = nil
|
85
|
+
|
86
|
+
exception = assert_raises(Exception) do
|
87
|
+
Stalking::Consumer.new do
|
88
|
+
job "test" do |args|
|
89
|
+
res = ["test", args]
|
90
|
+
|
91
|
+
raise Exception
|
92
|
+
end
|
93
|
+
end
|
94
|
+
end
|
95
|
+
|
96
|
+
assert_equal ["test", { "a" => "b" }], res
|
97
|
+
end
|
98
|
+
|
99
|
+
def test_work
|
100
|
+
Stalking::Producer.new.enqueue "test1", "a" => "b"
|
101
|
+
Stalking::Producer.new.enqueue "test2", "a" => "b"
|
102
|
+
Stalking::Producer.new.enqueue "test3", "a" => "b"
|
103
|
+
|
104
|
+
res = []
|
105
|
+
|
106
|
+
exception = assert_raises(Exception) do
|
107
|
+
Stalking::Consumer.new do
|
108
|
+
job "test1" do |args|
|
109
|
+
res.push "test1"
|
110
|
+
end
|
111
|
+
|
112
|
+
job "test2" do |args|
|
113
|
+
res.push "test2"
|
114
|
+
end
|
115
|
+
|
116
|
+
job "test3" do |args|
|
117
|
+
res.push "test3"
|
118
|
+
|
119
|
+
raise Exception
|
120
|
+
end
|
121
|
+
end
|
122
|
+
end
|
123
|
+
|
124
|
+
assert_equal ["test1", "test2", "test3"], res
|
125
|
+
end
|
126
|
+
|
127
|
+
def test_quit
|
128
|
+
raise NotImplementedError
|
129
|
+
end
|
130
|
+
|
131
|
+
def test_job_not_deleted
|
132
|
+
Stalking::Producer.new.enqueue "test", "a" => "b"
|
133
|
+
|
134
|
+
exception = assert_raises(Exception) do
|
135
|
+
Stalking::Consumer.new do
|
136
|
+
job "test" do |args|
|
137
|
+
raise Exception
|
138
|
+
end
|
139
|
+
end
|
140
|
+
end
|
141
|
+
|
142
|
+
res = nil
|
143
|
+
|
144
|
+
exception = assert_raises(Exception) do
|
145
|
+
Stalking::Consumer.new do
|
146
|
+
job "test" do |args|
|
147
|
+
res = ["test", args]
|
148
|
+
|
149
|
+
raise Exception
|
150
|
+
end
|
151
|
+
end
|
152
|
+
end
|
153
|
+
|
154
|
+
assert_equal ["test", {"a" => "b" }], res
|
155
|
+
end
|
156
|
+
|
157
|
+
def test_job_deleted
|
158
|
+
Stalking::Producer.new.enqueue "test", "a" => "b"
|
159
|
+
|
160
|
+
res = nil
|
161
|
+
|
162
|
+
exception = assert_raises(NoMethodError) do
|
163
|
+
Stalking::Consumer.new do
|
164
|
+
job "test" do |args|
|
165
|
+
res = ["test", args]
|
166
|
+
end
|
167
|
+
end
|
168
|
+
end
|
169
|
+
|
170
|
+
assert_equal ["test", { "a" => "b" }], res
|
171
|
+
assert_equal "undefined method `body' for nil:NilClass", exception.message
|
172
|
+
end
|
173
|
+
|
174
|
+
def test_reconnect
|
175
|
+
Stalking::Producer.new.enqueue "test", "a" => "b"
|
176
|
+
|
177
|
+
logger = TestLogger.new
|
178
|
+
|
179
|
+
res = []
|
180
|
+
|
181
|
+
exception = assert_raises(Exception) do
|
182
|
+
Stalking::Consumer.new(:logger => logger) do
|
183
|
+
job "test" do |args|
|
184
|
+
if res.empty?
|
185
|
+
res.push ["down"]
|
186
|
+
|
187
|
+
# Simulate a temporary beanstalk connection error.
|
188
|
+
|
189
|
+
Beanstalk::Pool.new(["localhost:11300"]).disconnect!
|
190
|
+
else
|
191
|
+
res.push ["test", args]
|
192
|
+
|
193
|
+
raise Exception
|
194
|
+
end
|
195
|
+
end
|
196
|
+
end
|
197
|
+
end
|
198
|
+
|
199
|
+
assert_equal [["down"], ["test", { "a" => "b" }]], res
|
200
|
+
|
201
|
+
assert_equal ["Stalking error: Beanstalk::NotConnected"], logger.messages
|
202
|
+
end
|
203
|
+
|
204
|
+
def test_logger
|
205
|
+
# Already tested
|
206
|
+
end
|
207
|
+
end
|
208
|
+
|
@@ -0,0 +1,86 @@
|
|
1
|
+
|
2
|
+
$:.unshift File.expand_path("../../lib", __FILE__)
|
3
|
+
|
4
|
+
require "stalking"
|
5
|
+
require "test/unit"
|
6
|
+
|
7
|
+
require File.expand_path("../test_logger", __FILE__)
|
8
|
+
|
9
|
+
class Stalking::ProducerTest < Test::Unit::TestCase
|
10
|
+
def setup
|
11
|
+
Beanstalk::Pool.clear!
|
12
|
+
end
|
13
|
+
|
14
|
+
def test_enqueue
|
15
|
+
Stalking::Producer.new.enqueue "test", { "key" => "value" }, :pri => 100, :delay => 100, :ttr => 100
|
16
|
+
|
17
|
+
assert_equal ['["test",{"key":"value"}]', 100, 100, 100], Beanstalk::Pool.new(["localhost:11300"]).queues["test"].last
|
18
|
+
end
|
19
|
+
|
20
|
+
def test_defaults
|
21
|
+
Stalking::Producer.new.enqueue "test", "key" => "value"
|
22
|
+
|
23
|
+
assert_equal ['["test",{"key":"value"}]', 65536, 0, 120], Beanstalk::Pool.new(["localhost:11300"]).queues["test"].last
|
24
|
+
end
|
25
|
+
|
26
|
+
def test_reconnect
|
27
|
+
producer = Stalking::Producer.new(:servers => ["server1", "server2"])
|
28
|
+
|
29
|
+
server1 = Beanstalk::Pool.new(["server1"])
|
30
|
+
server2 = Beanstalk::Pool.new(["server2"])
|
31
|
+
|
32
|
+
srand 0
|
33
|
+
|
34
|
+
producer.enqueue "test", "up" => "true"
|
35
|
+
|
36
|
+
assert_equal ['["test",{"up":"true"}]', 65536, 0, 120], server1.queues["test"].last
|
37
|
+
|
38
|
+
srand 0
|
39
|
+
|
40
|
+
server1.disconnect!
|
41
|
+
|
42
|
+
producer.enqueue "test", "down" => "disconnected"
|
43
|
+
|
44
|
+
assert_equal ['["test",{"down":"disconnected"}]', 65536, 0, 120], server1.queues["test"].last
|
45
|
+
|
46
|
+
server1.down!
|
47
|
+
|
48
|
+
producer.enqueue "test", "down" => "permanent"
|
49
|
+
|
50
|
+
assert_equal ['["test",{"down":"permanent"}]', 65536, 0, 120], server2.queues["test"].last
|
51
|
+
end
|
52
|
+
|
53
|
+
def test_timeout
|
54
|
+
producer = Stalking::Producer.new(:servers => ["server1", "server2"])
|
55
|
+
|
56
|
+
server1 = Beanstalk::Pool.new(["server1"])
|
57
|
+
server2 = Beanstalk::Pool.new(["server2"])
|
58
|
+
|
59
|
+
srand 0
|
60
|
+
|
61
|
+
producer.enqueue "test", "up" => "true"
|
62
|
+
|
63
|
+
assert_equal ['["test",{"up":"true"}]', 65536, 0, 120], server1.queues["test"].last
|
64
|
+
|
65
|
+
srand 0
|
66
|
+
|
67
|
+
server1.delay = 5
|
68
|
+
|
69
|
+
producer.enqueue "test", "down" => "delayed"
|
70
|
+
|
71
|
+
assert_equal ['["test",{"down":"delayed"}]', 65536, 0, 120], server2.queues["test"].last
|
72
|
+
end
|
73
|
+
|
74
|
+
def test_logger
|
75
|
+
logger = TestLogger.new
|
76
|
+
|
77
|
+
producer = Stalking::Producer.new(:logger => logger)
|
78
|
+
|
79
|
+
Beanstalk::Pool.new(["localhost:11300"]).down!
|
80
|
+
|
81
|
+
producer.enqueue "test", "up" => "true"
|
82
|
+
|
83
|
+
assert_equal ["Stalking error: Beanstalk::NotConnected"], logger.messages
|
84
|
+
end
|
85
|
+
end
|
86
|
+
|
metadata
ADDED
@@ -0,0 +1,111 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: stalking
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.0.1
|
5
|
+
prerelease:
|
6
|
+
platform: ruby
|
7
|
+
authors:
|
8
|
+
- Benjamin Vetter
|
9
|
+
autorequire:
|
10
|
+
bindir: bin
|
11
|
+
cert_chain: []
|
12
|
+
date: 2013-06-28 00:00:00.000000000 Z
|
13
|
+
dependencies:
|
14
|
+
- !ruby/object:Gem::Dependency
|
15
|
+
name: rake
|
16
|
+
requirement: !ruby/object:Gem::Requirement
|
17
|
+
none: false
|
18
|
+
requirements:
|
19
|
+
- - ! '>='
|
20
|
+
- !ruby/object:Gem::Version
|
21
|
+
version: '0'
|
22
|
+
type: :development
|
23
|
+
prerelease: false
|
24
|
+
version_requirements: !ruby/object:Gem::Requirement
|
25
|
+
none: false
|
26
|
+
requirements:
|
27
|
+
- - ! '>='
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
version: '0'
|
30
|
+
- !ruby/object:Gem::Dependency
|
31
|
+
name: beanstalk-client
|
32
|
+
requirement: !ruby/object:Gem::Requirement
|
33
|
+
none: false
|
34
|
+
requirements:
|
35
|
+
- - ! '>='
|
36
|
+
- !ruby/object:Gem::Version
|
37
|
+
version: '0'
|
38
|
+
type: :runtime
|
39
|
+
prerelease: false
|
40
|
+
version_requirements: !ruby/object:Gem::Requirement
|
41
|
+
none: false
|
42
|
+
requirements:
|
43
|
+
- - ! '>='
|
44
|
+
- !ruby/object:Gem::Version
|
45
|
+
version: '0'
|
46
|
+
- !ruby/object:Gem::Dependency
|
47
|
+
name: json_pure
|
48
|
+
requirement: !ruby/object:Gem::Requirement
|
49
|
+
none: false
|
50
|
+
requirements:
|
51
|
+
- - ! '>='
|
52
|
+
- !ruby/object:Gem::Version
|
53
|
+
version: '0'
|
54
|
+
type: :runtime
|
55
|
+
prerelease: false
|
56
|
+
version_requirements: !ruby/object:Gem::Requirement
|
57
|
+
none: false
|
58
|
+
requirements:
|
59
|
+
- - ! '>='
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '0'
|
62
|
+
description: Asynchronous job processing using the beanstalkd queue server with automatic
|
63
|
+
reconnects and failover
|
64
|
+
email:
|
65
|
+
- vetter@flakks.com
|
66
|
+
executables:
|
67
|
+
- stalking
|
68
|
+
extensions: []
|
69
|
+
extra_rdoc_files: []
|
70
|
+
files:
|
71
|
+
- .gitignore
|
72
|
+
- Gemfile
|
73
|
+
- LICENSE.txt
|
74
|
+
- README.md
|
75
|
+
- Rakefile
|
76
|
+
- bin/stalking
|
77
|
+
- lib/stalking.rb
|
78
|
+
- lib/stalking/consumer.rb
|
79
|
+
- lib/stalking/handlers.rb
|
80
|
+
- lib/stalking/producer.rb
|
81
|
+
- lib/stalking/version.rb
|
82
|
+
- stalking.gemspec
|
83
|
+
- test/stalking/beanstalk.rb
|
84
|
+
- test/stalking/consumer_test.rb
|
85
|
+
- test/stalking/producer_test.rb
|
86
|
+
- test/stalking/test_logger.rb
|
87
|
+
homepage: https://github.com/mrkamel/stalking
|
88
|
+
licenses: []
|
89
|
+
post_install_message:
|
90
|
+
rdoc_options: []
|
91
|
+
require_paths:
|
92
|
+
- lib
|
93
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
94
|
+
none: false
|
95
|
+
requirements:
|
96
|
+
- - ! '>='
|
97
|
+
- !ruby/object:Gem::Version
|
98
|
+
version: '0'
|
99
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
100
|
+
none: false
|
101
|
+
requirements:
|
102
|
+
- - ! '>='
|
103
|
+
- !ruby/object:Gem::Version
|
104
|
+
version: '0'
|
105
|
+
requirements: []
|
106
|
+
rubyforge_project:
|
107
|
+
rubygems_version: 1.8.23
|
108
|
+
signing_key:
|
109
|
+
specification_version: 3
|
110
|
+
summary: Asynchronous jobs using beanstalkd with automatic reconnects and failover
|
111
|
+
test_files: []
|