beanpicker 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- data/.rspec +2 -0
- data/README.markdown +186 -0
- data/Rakefile +47 -0
- data/beanpicker.gemspec +61 -0
- data/bin/combine +88 -0
- data/examples/sandwich.rb +46 -0
- data/lib/beanpicker.rb +268 -0
- data/lib/beanpicker/job_server.rb +374 -0
- data/lib/beanpicker/process.rb +23 -0
- data/lib/beanpicker/version.rb +9 -0
- data/spec/beanpicker_spec.rb +478 -0
- data/spec/spec_helper.rb +168 -0
- metadata +103 -0
data/.rspec
ADDED
data/README.markdown
ADDED
@@ -0,0 +1,186 @@
|
|
1
|
+
# Beanpicker
|
2
|
+
|
3
|
+
## What is it?
|
4
|
+
|
5
|
+
Beanpicker is a job queueing DSL for Beanstalk
|
6
|
+
|
7
|
+
## What? Beanstalk? It make coffe?
|
8
|
+
|
9
|
+
[beanstalk(d)][beanstalk] is a fast, lightweight queueing backend inspired by memcached. The Ruby Beanstalk client is a bit raw, however, so Beanpicker provides a thin wrapper to make job queueing from your Ruby app easy and fun.
|
10
|
+
|
11
|
+
## Is this similar to Stalker?
|
12
|
+
|
13
|
+
Yes, it is inspired in [stalker][stalker] and in [Minion][minion]
|
14
|
+
|
15
|
+
### Why should I use Beanpicker instead of Stalker?
|
16
|
+
|
17
|
+
Beanpicker work with subprocess. It create a fork for every request and destroy it in the end.
|
18
|
+
|
19
|
+
The vantages of Beanpicker are:
|
20
|
+
|
21
|
+
* Your job can use a large amount of RAM, it will be discarted when it end
|
22
|
+
* You can change vars in jobs, they will not be changed to other jobs nor the principal process
|
23
|
+
* If all your jobs need a large lib to be loaded(Rails?), you load it once in principal process and it will be available to all jobs without ocupping extra RAM
|
24
|
+
* It use YAML instead of JSON, so you can pass certain Ruby Objects(Time, Date, etc) //Please, don't try to pass a Rails Model or anything similar, pass only the ID so the job can avoid a possible outdated object
|
25
|
+
|
26
|
+
### How is his performance?
|
27
|
+
|
28
|
+
My machine:
|
29
|
+
|
30
|
+
* Notebook LG 590 5100
|
31
|
+
* Processor Intel Core i3 330M
|
32
|
+
* 4GB RAM DDR3
|
33
|
+
* Arch Linux x86-64
|
34
|
+
|
35
|
+
The speed with 10000 requests given 'with fork' is :fork => :every and 'without fork' is :fork => :master/false
|
36
|
+
|
37
|
+
\# time / requests per second / cpu load
|
38
|
+
|
39
|
+
* MRI 1.9.2
|
40
|
+
* With fork: 117.85 / 84.85 / 100%
|
41
|
+
* Without fork: 4.14 / 2415 / 30%
|
42
|
+
* MRI 1.8.7
|
43
|
+
* With fork: 122.27 / 81.78 / 58%
|
44
|
+
* Without fork: 6.34 / 1577 / 30~40%
|
45
|
+
* REE 1.8.7
|
46
|
+
* With fork: 121.92 / 82.02 / 20~60%
|
47
|
+
* Without fork: 4.77 / 2096 / 30%
|
48
|
+
* JRuby 1.5.6 + OpenJDK 6.2b0\_1.9.3 VM 1.6.0\_20
|
49
|
+
* With fork: don't accept fork?
|
50
|
+
* Without fork: 10.99 / 909.91 / 97%
|
51
|
+
* Rubinius 1.2.0
|
52
|
+
* With fork: don't try, too much errors
|
53
|
+
* Without fork: 11.24 / 889 / 36~52%
|
54
|
+
|
55
|
+
Fork is activated by default, it should slow down your application but keep safe from memory leaks.
|
56
|
+
|
57
|
+
You can easy active or desactive the fork for a job with:
|
58
|
+
|
59
|
+
job "job.without.fork", :fork => false do |args|
|
60
|
+
debug "Running on a thread in main process"
|
61
|
+
warn "This process will grow because of any job running on main process"
|
62
|
+
end
|
63
|
+
|
64
|
+
job "job.with.fork.every.time", :fork => :every do |args|
|
65
|
+
debug "Running on a fork of main process"
|
66
|
+
debug "This process will be killed on end of this job"
|
67
|
+
debug "This decrease the peformance but save from memory leaks"
|
68
|
+
debug "All extra memory used by this process will vanish in end"
|
69
|
+
end
|
70
|
+
|
71
|
+
job "job.with.fork.once", :fork => :master do |args|
|
72
|
+
debug "Running on a fork of main process"
|
73
|
+
debug "This process will not be killed on end of this job"
|
74
|
+
debug "This increase the performance but don't save from memory leaks"
|
75
|
+
debug "This process will only grow in memory because of code executed in 'job.with.fork.once'"
|
76
|
+
end
|
77
|
+
|
78
|
+
You can pass :fork\_every => true(default)/false and :fork\_master => true/false(default)
|
79
|
+
|
80
|
+
The :fork argument overwrite :fork\_every and :fork\_master
|
81
|
+
|
82
|
+
The default :fork\_every and :fork\_master are setted on Beanpicker::default\_fork\_[master|every]
|
83
|
+
|
84
|
+
Beanpicker::fork\_every and Beanpicker::fork\_master overwrite the job options, so, if you set they false the jobs will run in the main thread even if they specify the :fork, :fork\_every and/or :fork\_master
|
85
|
+
|
86
|
+
## Queueing jobs
|
87
|
+
|
88
|
+
From anywhere in your app:
|
89
|
+
|
90
|
+
require 'beanpicker'
|
91
|
+
|
92
|
+
Beanpicker.enqueue('email.send', :to => 'joe@example.com')
|
93
|
+
Beanpicker.enqueue('post.cleanup.all')
|
94
|
+
Beanpicker.enqueue('post.cleanup', :id => post.id)
|
95
|
+
|
96
|
+
### Chain jobs
|
97
|
+
|
98
|
+
If you have a task that requires more than one step just pass an array of queues when you enqueue.
|
99
|
+
|
100
|
+
require 'beanpicker/job_server'
|
101
|
+
|
102
|
+
Beanpicker::Worker.new do
|
103
|
+
# this is a slow job, so we'll spawn 10 forks of it :)
|
104
|
+
job "email.fetch_attachments", :childs => 10 do |args|
|
105
|
+
attachment_ids = Email.fetch_attachments_for args[:email_id]
|
106
|
+
{ :attachment_ids => attachment_ids }
|
107
|
+
end
|
108
|
+
|
109
|
+
# by default :childs is 1
|
110
|
+
job "email.send" do |args|
|
111
|
+
Email.send({
|
112
|
+
:id => args[:email_id],
|
113
|
+
:attachments => args[:attachment_ids].map { |a| Attachment.find(a) }
|
114
|
+
})
|
115
|
+
end
|
116
|
+
|
117
|
+
end
|
118
|
+
|
119
|
+
Beanpicker.enqueue(["email.fetch_attachments", "email.send"], :email_id => 10)
|
120
|
+
|
121
|
+
|
122
|
+
## Output Messages
|
123
|
+
|
124
|
+
Inside of a job you can use debug, info, warn, error and fatal. It will be redirected to logger(STDOUT by default)
|
125
|
+
|
126
|
+
## Options
|
127
|
+
|
128
|
+
### Global options
|
129
|
+
|
130
|
+
All options are inside of module Beanpicker
|
131
|
+
|
132
|
+
* Used for jobs:
|
133
|
+
* Global:
|
134
|
+
* default\_fork\_every: If should fork a job and destroy every time It will run. This options is overwrited by specified job options. Default true
|
135
|
+
* default\_fork\_master: If should fork the child process. This options is overwrited by specified job options. Default false
|
136
|
+
* fork\_every: Like default\_fork\_every, but overwrite job options. Default nil
|
137
|
+
* fork\_master: Like default\_fork\_master, but overwrite job options. Default nil
|
138
|
+
* default\_childs\_number: How much childs should be started for every job? Default 1
|
139
|
+
* In job file(not function):
|
140
|
+
* log\_file : Use a own log file, this file should be used to all jobs, except the ones who specify a :log\_file
|
141
|
+
* In 'job' function:
|
142
|
+
* :fork\_every : Overwrite default\_fork\_every and is overwrited by fork\_every
|
143
|
+
* :fork\_master : Overwrite default\_fork\_master and is overwrited by fork\_master
|
144
|
+
* :fork : Overwrite :fork\_every and :fork\_master, expect :every to every=true and master=false, :master to every=false and master=true or other value(any) to every=false and master=false. The result can be overwrited by fork\_master and fork\_every.
|
145
|
+
* :log\_file : Use a own log file
|
146
|
+
* Used for enqueue
|
147
|
+
* Global
|
148
|
+
* default\_pri: The priority of job. Default 65536
|
149
|
+
* default\_delay: The delay to start the job. Default 0
|
150
|
+
* default\_ttr: The time to run the job. Default is 120
|
151
|
+
* In 'enqueue' function
|
152
|
+
* :pri
|
153
|
+
* :delay
|
154
|
+
* :ttr
|
155
|
+
|
156
|
+
|
157
|
+
## Using combine
|
158
|
+
|
159
|
+
Beanpicker ships with "combine", "A Beanpicker server"
|
160
|
+
|
161
|
+
Try combine --help to see all options
|
162
|
+
|
163
|
+
e.g. command:
|
164
|
+
|
165
|
+
combine -r config.rb -l log/jobs.log sandwich_jobs.rb email_jobs.rb
|
166
|
+
|
167
|
+
|
168
|
+
## Multiple Beanstalk servers
|
169
|
+
|
170
|
+
Beanpicker look in ENV variables BEANSTALK\_URL and BEANSTALK\_URLS.
|
171
|
+
|
172
|
+
In BEANSTALK\_URL it expect a url like "server[:port]" or "beanstalk://server[:port]".
|
173
|
+
|
174
|
+
In BEANSTALK\_URLS it expect a list of urls separed by comma. e.g. "localhost,localhost:11301,10.1.1.9,10.1.1.10:3000"
|
175
|
+
|
176
|
+
## Credits
|
177
|
+
|
178
|
+
Created by [Renan Fernandes][renan-website]
|
179
|
+
|
180
|
+
Released under the [MIT License][license]
|
181
|
+
|
182
|
+
[beanstalk]: http://kr.github.com/beanstalkd/ "Beanstalk"
|
183
|
+
[stalker]: http://github.com/adamwiggins/stalker "Stalker"
|
184
|
+
[minion]: http://github.com/orionz/minion "Minion"
|
185
|
+
[license]: http://www.opensource.org/licenses/mit-license.php "MIT License"
|
186
|
+
[renan-website]: http://renanfernandes.com.br "Author's Website"
|
data/Rakefile
ADDED
@@ -0,0 +1,47 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'rake'
|
3
|
+
|
4
|
+
require File.expand_path("../lib/beanpicker/version", __FILE__)
|
5
|
+
|
6
|
+
begin
|
7
|
+
require 'jeweler'
|
8
|
+
Jeweler::Tasks.new do |gem|
|
9
|
+
gem.name = "beanpicker"
|
10
|
+
gem.summary = "DSL for beanstalkd, similar to Stalker/Minion"
|
11
|
+
gem.description = "DSL for beanstalkd, similar to Stalker/Minion but uses subprocesses"
|
12
|
+
gem.email = "renan@kauamanga.com.br"
|
13
|
+
gem.homepage = "http://github.com/ShadowBelmolve/beanpicker"
|
14
|
+
gem.authors = ["Renan Fernandes"]
|
15
|
+
gem.license = "MIT"
|
16
|
+
gem.version = Beanpicker::VERSION_STRING
|
17
|
+
gem.add_dependency "beanstalk-client"
|
18
|
+
gem.add_development_dependency "rspec", ">= 2.0"
|
19
|
+
gem.executables = ["combine"]
|
20
|
+
|
21
|
+
|
22
|
+
end
|
23
|
+
Jeweler::GemcutterTasks.new
|
24
|
+
Jeweler::RubygemsDotOrgTasks.new
|
25
|
+
rescue LoadError
|
26
|
+
puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
|
27
|
+
end
|
28
|
+
|
29
|
+
|
30
|
+
desc "Run the tests with RSpec"
|
31
|
+
task :test do
|
32
|
+
require 'rspec/autorun'
|
33
|
+
ARGV.clear
|
34
|
+
ARGV << "spec"
|
35
|
+
end
|
36
|
+
|
37
|
+
task :default => :test
|
38
|
+
|
39
|
+
require 'rake/rdoctask'
|
40
|
+
Rake::RDocTask.new do |rdoc|
|
41
|
+
|
42
|
+
rdoc.rdoc_dir = 'doc'
|
43
|
+
rdoc.title = "Beanpicker #{Beanpicker::VERSION_STRING}"
|
44
|
+
rdoc.rdoc_files.include('README*')
|
45
|
+
rdoc.rdoc_files.include('lib/**/*.rb')
|
46
|
+
end
|
47
|
+
|
data/beanpicker.gemspec
ADDED
@@ -0,0 +1,61 @@
|
|
1
|
+
# Generated by jeweler
|
2
|
+
# DO NOT EDIT THIS FILE DIRECTLY
|
3
|
+
# Instead, edit Jeweler::Tasks in Rakefile, and run 'rake gemspec'
|
4
|
+
# -*- encoding: utf-8 -*-
|
5
|
+
|
6
|
+
Gem::Specification.new do |s|
|
7
|
+
s.name = %q{beanpicker}
|
8
|
+
s.version = "0.1.0"
|
9
|
+
|
10
|
+
s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
|
11
|
+
s.authors = ["Renan Fernandes"]
|
12
|
+
s.date = %q{2010-12-28}
|
13
|
+
s.default_executable = %q{combine}
|
14
|
+
s.description = %q{DSL for beanstalkd, similar to Stalker/Minion but uses subprocesses}
|
15
|
+
s.email = %q{renan@kauamanga.com.br}
|
16
|
+
s.executables = ["combine"]
|
17
|
+
s.extra_rdoc_files = [
|
18
|
+
"README.markdown"
|
19
|
+
]
|
20
|
+
s.files = [
|
21
|
+
".rspec",
|
22
|
+
"README.markdown",
|
23
|
+
"Rakefile",
|
24
|
+
"beanpicker.gemspec",
|
25
|
+
"bin/combine",
|
26
|
+
"examples/sandwich.rb",
|
27
|
+
"lib/beanpicker.rb",
|
28
|
+
"lib/beanpicker/job_server.rb",
|
29
|
+
"lib/beanpicker/process.rb",
|
30
|
+
"lib/beanpicker/version.rb",
|
31
|
+
"spec/beanpicker_spec.rb",
|
32
|
+
"spec/spec_helper.rb"
|
33
|
+
]
|
34
|
+
s.homepage = %q{http://github.com/ShadowBelmolve/beanpicker}
|
35
|
+
s.licenses = ["MIT"]
|
36
|
+
s.require_paths = ["lib"]
|
37
|
+
s.rubygems_version = %q{1.3.7}
|
38
|
+
s.summary = %q{DSL for beanstalkd, similar to Stalker/Minion}
|
39
|
+
s.test_files = [
|
40
|
+
"examples/sandwich.rb",
|
41
|
+
"spec/beanpicker_spec.rb",
|
42
|
+
"spec/spec_helper.rb"
|
43
|
+
]
|
44
|
+
|
45
|
+
if s.respond_to? :specification_version then
|
46
|
+
current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
|
47
|
+
s.specification_version = 3
|
48
|
+
|
49
|
+
if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
|
50
|
+
s.add_runtime_dependency(%q<beanstalk-client>, [">= 0"])
|
51
|
+
s.add_development_dependency(%q<rspec>, [">= 2.0"])
|
52
|
+
else
|
53
|
+
s.add_dependency(%q<beanstalk-client>, [">= 0"])
|
54
|
+
s.add_dependency(%q<rspec>, [">= 2.0"])
|
55
|
+
end
|
56
|
+
else
|
57
|
+
s.add_dependency(%q<beanstalk-client>, [">= 0"])
|
58
|
+
s.add_dependency(%q<rspec>, [">= 2.0"])
|
59
|
+
end
|
60
|
+
end
|
61
|
+
|
data/bin/combine
ADDED
@@ -0,0 +1,88 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
|
3
|
+
require 'optparse'
|
4
|
+
require File.expand_path( File.join(File.dirname(__FILE__), "..", "lib", "beanpicker", "job_server") )
|
5
|
+
|
6
|
+
_opts = nil
|
7
|
+
|
8
|
+
parser = OptionParser.new do |opts|
|
9
|
+
_opts = opts
|
10
|
+
|
11
|
+
opts.banner = 'combine is a beanpicker server'
|
12
|
+
opts.separator ''
|
13
|
+
opts.separator "e.g. #{$0} -r default.rb -em -l log/out.log sandwhich_jobs.rb email_jobs.rb"
|
14
|
+
opts.separator ''
|
15
|
+
|
16
|
+
opts.on('-r', '--require FILE', "Require a file to be loaded before the workers") do |f|
|
17
|
+
require File.expand_path(f)
|
18
|
+
end
|
19
|
+
|
20
|
+
opts.on('-e', '--[no-]fork-every', "Run every job in their own process?",
|
21
|
+
"This option overwrite the options of jobs") do |e|
|
22
|
+
Beanpicker::fork_every = e
|
23
|
+
end
|
24
|
+
|
25
|
+
opts.on('-m', '--[no-]fork-master', "Run every child in their own process?",
|
26
|
+
"This option overwrite the options of jobs") do |m|
|
27
|
+
Beanpicker::fork_master = m
|
28
|
+
end
|
29
|
+
|
30
|
+
opts.on('-y', '--[no-]fork-every-default', "Same of -e, but don't overwrite jobs options",
|
31
|
+
"Default is true") do |y|
|
32
|
+
Beanpicker::default_fork_every = y
|
33
|
+
end
|
34
|
+
|
35
|
+
opts.on('-a', '--[no-]fork-master-default', "Same of -m, but don't overwrite jobs options",
|
36
|
+
"Default is false") do |a|
|
37
|
+
Beanpicker::default_fork_master = a
|
38
|
+
end
|
39
|
+
|
40
|
+
opts.on('-c', '--childs-number-default N', Integer, "The number of childs every job should have",
|
41
|
+
"Default is 1") do |n|
|
42
|
+
Beanpicker::default_childs_number = n > 1 ? n : 1
|
43
|
+
end
|
44
|
+
|
45
|
+
opts.on('-l', '--logger [FILE]', "Redirect messages to file", "Default is STDOUT") do |f|
|
46
|
+
Beanpicker::log_handler = f || STDOUT
|
47
|
+
end
|
48
|
+
|
49
|
+
opts.on_tail('-h', '--help', "Show this message and exit") do
|
50
|
+
puts opts
|
51
|
+
exit
|
52
|
+
end
|
53
|
+
|
54
|
+
opts.on_tail('-v', '--version', "Show the version and exit") do
|
55
|
+
puts "beanpicker version: #{Beanpicker::VERSION_STRING}"
|
56
|
+
exit
|
57
|
+
end
|
58
|
+
|
59
|
+
end
|
60
|
+
|
61
|
+
begin
|
62
|
+
parser.parse!(ARGV)
|
63
|
+
rescue => e
|
64
|
+
STDERR.puts "Error when parsing options.\n#{e.message}\n\n"
|
65
|
+
puts _opts
|
66
|
+
exit
|
67
|
+
end
|
68
|
+
|
69
|
+
if ARGV.empty?
|
70
|
+
puts "You should specify at least a file with jobs to run"
|
71
|
+
puts _opts
|
72
|
+
exit
|
73
|
+
end
|
74
|
+
|
75
|
+
for file in ARGV
|
76
|
+
if File.exists?(file)
|
77
|
+
if not File.readable?(file)
|
78
|
+
puts "File #{file} exits but isn't readable!"
|
79
|
+
exit
|
80
|
+
end
|
81
|
+
else
|
82
|
+
puts "File #{file} don't exists!"
|
83
|
+
exit
|
84
|
+
end
|
85
|
+
end
|
86
|
+
|
87
|
+
server = Beanpicker::Server.new(ARGV)
|
88
|
+
server.run
|
@@ -0,0 +1,46 @@
|
|
1
|
+
#example of jobs
|
2
|
+
|
3
|
+
|
4
|
+
# It's a slow job, so we'll spawn 10 forks
|
5
|
+
# It'll be forked every time
|
6
|
+
job "sandwich.make", :childs => 10, :fork => :every do |args|
|
7
|
+
debug "Making a sandwich to #{args[:for]}"
|
8
|
+
# very slow job
|
9
|
+
id = 10
|
10
|
+
{ :id => id }
|
11
|
+
end
|
12
|
+
|
13
|
+
# This is as fast job and don't make memory leaks, so We'll fork once
|
14
|
+
job "sandwich.sell", :fork => :master do |args|
|
15
|
+
debug "Selling the sandwich #{args[:id]} to #{args[:for]}"
|
16
|
+
#client = Client.find_by_name(args[:for])
|
17
|
+
#sandwich = Sandwich.find(args[:id])
|
18
|
+
#sale = client.buy_sandwich sandwich
|
19
|
+
sale = args[:for] == "Renan"
|
20
|
+
if sale
|
21
|
+
{ :sale_id => 20 }
|
22
|
+
else
|
23
|
+
warn "Can't sell the sandwich to #{args[:for]} :("
|
24
|
+
false
|
25
|
+
end
|
26
|
+
end
|
27
|
+
|
28
|
+
# fast job but make memory leaks, so We'll fork every time
|
29
|
+
job "sandwich.ingredients.recalcule", :childs => 3, :fork => :every do |args|
|
30
|
+
debug "Recalculating the ingredients of sandwich #{args[:id]} sold to #{args[:for]} in sale #{args[:sale_id]}"
|
31
|
+
#Ingredients.recalcule_based_on_sale(args[:sale_id])
|
32
|
+
end
|
33
|
+
|
34
|
+
# Example of Normal Jobs
|
35
|
+
# def make_sandwich_for(client_name)
|
36
|
+
# Beanpicker.enqueue("sandwich.make", :for => client_name)
|
37
|
+
# end
|
38
|
+
# This will call the "sandwich.make" job and pass a hash with client_name on :for key
|
39
|
+
|
40
|
+
# Example of Chain Jobs
|
41
|
+
# Beanpicker.enqueue(["sandwich.make", "sandwich.sell", "sandwich.ingredients.recalcule"], :for => "Renan")
|
42
|
+
# This will call all the three jobs
|
43
|
+
#
|
44
|
+
# Beanpicker.enqueue(["sandwich.make", "sandwich.sell", "sandwich.ingredients.recalcule"], :for => "Raphael")
|
45
|
+
# This will call the two first jobs, but not the third. Raphael don't have money to buy a sandwich, so the
|
46
|
+
# second job will return false and Beanpicker will break the chain
|
data/lib/beanpicker.rb
ADDED
@@ -0,0 +1,268 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'beanstalk-client'
|
3
|
+
require 'uri'
|
4
|
+
require 'logger'
|
5
|
+
|
6
|
+
$:.unshift( File.expand_path(File.dirname(__FILE__)) )
|
7
|
+
|
8
|
+
require 'beanpicker/version'
|
9
|
+
|
10
|
+
# The fucking master job DSL to beanstalkd
|
11
|
+
#
|
12
|
+
# Just use it and go to beach ;)
|
13
|
+
module Beanpicker
|
14
|
+
|
15
|
+
extend self
|
16
|
+
|
17
|
+
# Abstract logger methods
|
18
|
+
module MsgLogger
|
19
|
+
|
20
|
+
# call .debug of logger
|
21
|
+
def debug(m)
|
22
|
+
log_handler.debug msg(m)
|
23
|
+
end
|
24
|
+
|
25
|
+
# call .info of logger
|
26
|
+
def info(m)
|
27
|
+
log_handler.info msg(m)
|
28
|
+
end
|
29
|
+
|
30
|
+
# call .warn of logger
|
31
|
+
def warn(m)
|
32
|
+
log_handler.warn msg(m)
|
33
|
+
end
|
34
|
+
|
35
|
+
# call .error of logger
|
36
|
+
def error(m)
|
37
|
+
log_handler.error msg(m)
|
38
|
+
end
|
39
|
+
|
40
|
+
# call .fatal of logger
|
41
|
+
def fatal(m)
|
42
|
+
log_handler.fatal msg(m)
|
43
|
+
end
|
44
|
+
|
45
|
+
# prepare the message for logger
|
46
|
+
def msg(msg)
|
47
|
+
if @name
|
48
|
+
"[#{name}] #{msg}"
|
49
|
+
else
|
50
|
+
msg
|
51
|
+
end
|
52
|
+
end
|
53
|
+
|
54
|
+
# return the current logger os create a new
|
55
|
+
def log_handler
|
56
|
+
@log_handler ||= ::Logger.new(STDOUT)
|
57
|
+
end
|
58
|
+
|
59
|
+
# set a new logger
|
60
|
+
# if the argument is a String/IO it will create a new instance of Logger using it argument
|
61
|
+
# else it will see if the argument respond to debug, info, warn, error and fatal
|
62
|
+
def log_handler=(v)
|
63
|
+
if [String, IO].include?(v.class)
|
64
|
+
@log_handler = ::Logger.new(v)
|
65
|
+
else
|
66
|
+
for m in [:debug, :info, :warn, :error, :fatal]
|
67
|
+
unless v.respond_to?(m)
|
68
|
+
error "Logger #{v} don't respond to #{m}. Aborting!"
|
69
|
+
return
|
70
|
+
end
|
71
|
+
end
|
72
|
+
@log_handler = v
|
73
|
+
end
|
74
|
+
end
|
75
|
+
|
76
|
+
end
|
77
|
+
|
78
|
+
extend MsgLogger
|
79
|
+
|
80
|
+
# Send a new queue to beanstalkd
|
81
|
+
#
|
82
|
+
# The first argument is a String with the name of job or a Array of Strings to do job chains
|
83
|
+
#
|
84
|
+
# The second argument should be any object that will be passed in a YAML format to the job
|
85
|
+
#
|
86
|
+
# The third argument should be a hash containing :pri(priority) => Integer, :delay => Integer and :ttr(time-to-work) => Integer
|
87
|
+
#
|
88
|
+
# If beanstalk raise a Beanstalk::NotConnected, enqueue will create a new instance of beanstalk connection and retry.
|
89
|
+
# If it raise again, enqueue will raise the error
|
90
|
+
def enqueue(jobs, args={}, o={})
|
91
|
+
opts = [
|
92
|
+
o[:pri] || default_pri,
|
93
|
+
o[:delay] || default_delay,
|
94
|
+
o[:ttr] || default_ttr
|
95
|
+
]
|
96
|
+
|
97
|
+
jobs = [jobs.to_s] unless jobs.is_a?(Array)
|
98
|
+
jobs.compact!
|
99
|
+
raise ArgumentError, "you need at least 1 job" if jobs.empty?
|
100
|
+
job = jobs.first
|
101
|
+
|
102
|
+
beanstalk.use(job)
|
103
|
+
beanstalk.yput({ :args => args, :next_jobs => jobs[1..-1]}, *opts)
|
104
|
+
rescue Beanstalk::NotConnected => e
|
105
|
+
raise e if defined?(r)
|
106
|
+
r = true
|
107
|
+
error exception_message(e, "You have a problem with beanstalkd.\nIs it running?")
|
108
|
+
@@beanstalk = new_beanstalk
|
109
|
+
retry
|
110
|
+
end
|
111
|
+
|
112
|
+
# Return the default beanstalk connection os create a new one with new_beanstalk
|
113
|
+
def beanstalk
|
114
|
+
@@beanstalk ||= new_beanstalk
|
115
|
+
end
|
116
|
+
|
117
|
+
# Create a new beanstalk connection using the urls from beanstalk_urls
|
118
|
+
def new_beanstalk
|
119
|
+
Beanstalk::Pool.new(beanstalk_urls)
|
120
|
+
end
|
121
|
+
|
122
|
+
# Look in ENV['BEANSTALK_URL'] and ENV['BEANSTALK_URLS'] for beanstalk urls and process returning a array.
|
123
|
+
#
|
124
|
+
# If don't find a good url it will return a array with just "localhost:11300"(default beanstalk port)
|
125
|
+
def beanstalk_urls
|
126
|
+
urls = [ENV['BEANSTALK_URL'], ENV['BEANSTALK_URLS']].compact.join(",").split(",").map do |url|
|
127
|
+
if url =~ /^beanstalk:\/\//
|
128
|
+
uri = URI.parse(url)
|
129
|
+
url = "#{uri.host}:#{uri.port}"
|
130
|
+
else
|
131
|
+
url = url.gsub(/^([^:\/]+)(:(\d+)).*$/) { "#{$1}:#{$3 || 11300}" }
|
132
|
+
end
|
133
|
+
url =~ /^[^:\/]+:\d+$/ ? url : nil
|
134
|
+
end.compact
|
135
|
+
urls.empty? ? ["localhost:11300"] : urls
|
136
|
+
end
|
137
|
+
|
138
|
+
# Helper to should a exception message
|
139
|
+
def exception_message(e, msg=nil)
|
140
|
+
m = []
|
141
|
+
m << msg if msg
|
142
|
+
m << e.message
|
143
|
+
m += e.backtrace
|
144
|
+
m.join("\n")
|
145
|
+
end
|
146
|
+
|
147
|
+
# Return the default priority(65536 by default).
|
148
|
+
#
|
149
|
+
# This is used by enqueue
|
150
|
+
def default_pri
|
151
|
+
@@default_pri ||= 65536
|
152
|
+
end
|
153
|
+
|
154
|
+
# Set the default priority
|
155
|
+
def default_pri=(v)
|
156
|
+
@@default_pri = v
|
157
|
+
end
|
158
|
+
|
159
|
+
# Return the default delay(0 by default)
|
160
|
+
#
|
161
|
+
# This is used by enqueue
|
162
|
+
def default_delay
|
163
|
+
@@default_delay ||= 0
|
164
|
+
end
|
165
|
+
|
166
|
+
# Set the default delay
|
167
|
+
def default_delay=(v)
|
168
|
+
@@default_delay = v
|
169
|
+
end
|
170
|
+
|
171
|
+
# Set the default time-to-work(120 by default)
|
172
|
+
#
|
173
|
+
# This is used by enqueue
|
174
|
+
def default_ttr
|
175
|
+
@@default_ttr ||= 120
|
176
|
+
end
|
177
|
+
|
178
|
+
# Set the default time-to-work
|
179
|
+
def default_ttr=(v)
|
180
|
+
@@default_ttr = v
|
181
|
+
end
|
182
|
+
|
183
|
+
# Return the default number of childs that a Worker should create(1 by default)
|
184
|
+
#
|
185
|
+
# This is used by Worker::Child::process
|
186
|
+
def default_childs_number
|
187
|
+
@@default_childs_number ||= 1
|
188
|
+
end
|
189
|
+
|
190
|
+
# Set the default childs number
|
191
|
+
def default_childs_number=(v)
|
192
|
+
@@default_childs_number = v
|
193
|
+
end
|
194
|
+
|
195
|
+
# Return if a child should fork every time that a job will process.
|
196
|
+
# This option is overwrited by job options and fork_every
|
197
|
+
#
|
198
|
+
# This is used by Worker::Child
|
199
|
+
def default_fork_every
|
200
|
+
defined?(@@default_fork_every) ? @@default_fork_every : true
|
201
|
+
end
|
202
|
+
|
203
|
+
# Set the default_fork_every
|
204
|
+
def default_fork_every=(v)
|
205
|
+
@@default_fork_every = !!v
|
206
|
+
end
|
207
|
+
|
208
|
+
# Return if a child should fork itself on intialize.
|
209
|
+
# This should be used when default_fork_every is false.
|
210
|
+
# This option is overwrited by job options and fork_master
|
211
|
+
#
|
212
|
+
# Use it only if the jobs need high speed and are "memory leak"-safe
|
213
|
+
#
|
214
|
+
# This is used by Worker::Child
|
215
|
+
def default_fork_master
|
216
|
+
defined?(@@default_fork_master) ? @@default_fork_master : false
|
217
|
+
end
|
218
|
+
|
219
|
+
# Set the default_fork_master
|
220
|
+
def default_fork_master=(v)
|
221
|
+
@@default_fork_master = !!v
|
222
|
+
end
|
223
|
+
|
224
|
+
# See default_fork_every
|
225
|
+
#
|
226
|
+
# This option overwrite all others
|
227
|
+
def fork_every
|
228
|
+
defined?(@@fork_every) ? @@fork_every : nil
|
229
|
+
end
|
230
|
+
|
231
|
+
# Set the fork_every
|
232
|
+
def fork_every=(v)
|
233
|
+
@@fork_every = v.nil? ? nil : !!v
|
234
|
+
end
|
235
|
+
|
236
|
+
# See default_fork_master
|
237
|
+
#
|
238
|
+
# This option overwrite all others
|
239
|
+
def fork_master
|
240
|
+
defined?(@@fork_master) ? @@fork_master : nil
|
241
|
+
end
|
242
|
+
|
243
|
+
# Set the fork_master
|
244
|
+
def fork_master=(v)
|
245
|
+
@@fork_master = v.nil? ? nil : !!v
|
246
|
+
end
|
247
|
+
|
248
|
+
# Return a Array with the workers registered
|
249
|
+
def workers
|
250
|
+
@@workers ||= []
|
251
|
+
end
|
252
|
+
|
253
|
+
# Add a worker to the list of workers
|
254
|
+
def add_worker(worker)
|
255
|
+
workers << worker
|
256
|
+
end
|
257
|
+
|
258
|
+
# Call die! for all childs of every worker and clear the list
|
259
|
+
# See workers
|
260
|
+
def stop_workers
|
261
|
+
for worker in workers
|
262
|
+
for child in worker.childs
|
263
|
+
child.die!
|
264
|
+
end
|
265
|
+
end
|
266
|
+
workers.clear
|
267
|
+
end
|
268
|
+
end
|