sweatshop 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/History.txt +6 -0
- data/LICENSE +20 -0
- data/README.markdown +103 -0
- data/Rakefile +43 -0
- data/VERSION.yml +4 -0
- data/bin/sweatd +3 -0
- data/config/defaults.yml +8 -0
- data/config/sweatshop.yml +15 -0
- data/install.rb +20 -0
- data/lib/message_queue/base.rb +17 -0
- data/lib/message_queue/kestrel.rb +32 -0
- data/lib/message_queue/rabbit.rb +108 -0
- data/lib/sweat_shop.rb +179 -0
- data/lib/sweat_shop/daemoned.rb +405 -0
- data/lib/sweat_shop/metaid.rb +5 -0
- data/lib/sweat_shop/sweatd.rb +76 -0
- data/lib/sweat_shop/worker.rb +162 -0
- data/script/initd.sh +108 -0
- data/script/kestrel +93 -0
- data/script/kestrel.sh +93 -0
- data/script/sweatshop +17 -0
- data/test/hello_worker.rb +13 -0
- data/test/test_functional_worker.rb +72 -0
- data/test/test_helper.rb +4 -0
- data/test/test_sweatshop.rb +78 -0
- metadata +91 -0
data/History.txt
ADDED
data/LICENSE
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
Copyright (c) 2009 Amos Elliston
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
4
|
+
a copy of this software and associated documentation files (the
|
5
|
+
"Software"), to deal in the Software without restriction, including
|
6
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
7
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
8
|
+
permit persons to whom the Software is furnished to do so, subject to
|
9
|
+
the following conditions:
|
10
|
+
|
11
|
+
The above copyright notice and this permission notice shall be
|
12
|
+
included in all copies or substantial portions of the Software.
|
13
|
+
|
14
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
15
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
16
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
17
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
18
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
19
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
20
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.markdown
ADDED
@@ -0,0 +1,103 @@
|
|
1
|
+
# SweatShop
|
2
|
+
|
3
|
+
SweatShop provides an api to background resource intensive tasks. Much of the api design was copied from Workling, with a few tweaks.
|
4
|
+
Currently, it runs rabbitmq and kestrel, but it can support any number of queues.
|
5
|
+
|
6
|
+
## Installing
|
7
|
+
|
8
|
+
gem install sweat_shop
|
9
|
+
freeze in your gems directory (add config.gem 'sweat_shop' to your environment)
|
10
|
+
cd vendor/gems/sweat_shop
|
11
|
+
rake setup
|
12
|
+
|
13
|
+
## Writing workers
|
14
|
+
|
15
|
+
Put `email_worker.rb` into app/workers and sublcass `SweatShop::Worker`:
|
16
|
+
|
17
|
+
class EmailWorker < SweatShop::Worker
|
18
|
+
def send_mail(to)
|
19
|
+
user = User.find_by_id(to)
|
20
|
+
Mailer.deliver_welcome(to)
|
21
|
+
end
|
22
|
+
end
|
23
|
+
|
24
|
+
Then, anywhere in your app you can execute:
|
25
|
+
|
26
|
+
EmailWorker.async_send_mail(1)
|
27
|
+
|
28
|
+
The `async` signifies that this task will be placed on a queue to be serviced by the EmailWorker possibly on another machine. You can also
|
29
|
+
call:
|
30
|
+
|
31
|
+
EmailWorker.send_mail(1)
|
32
|
+
|
33
|
+
That will do the work immediately, without placing the task on the queue. You can also define a `queue_group` at the top of the file
|
34
|
+
which will allow you to split workers out into logical groups. This is important if you have various machines serving different
|
35
|
+
queues.
|
36
|
+
|
37
|
+
## Running the queue
|
38
|
+
|
39
|
+
SweatShop has been tested with Rabbit and Kestrel, but it will also work with Starling. Please use the following resources to install the server:
|
40
|
+
|
41
|
+
Kestrel:
|
42
|
+
http://github.com/robey/kestrel/tree/master
|
43
|
+
|
44
|
+
Rabbit:
|
45
|
+
http://github.com/ezmobius/nanite/tree/master
|
46
|
+
|
47
|
+
config/sweatshop.yml specifies the machine address of the queue
|
48
|
+
(default localhost:5672). You can also specify the queue type with the
|
49
|
+
queue param.
|
50
|
+
|
51
|
+
## Rabbit cluster support
|
52
|
+
|
53
|
+
The following example configuration shows support for Rabbit clusters
|
54
|
+
within a queue group:
|
55
|
+
|
56
|
+
default:
|
57
|
+
queue: rabbit
|
58
|
+
cluster:
|
59
|
+
- hostA:5672
|
60
|
+
- hostB:5672
|
61
|
+
user: 'guest'
|
62
|
+
pass: 'guest'
|
63
|
+
vhost: '/'
|
64
|
+
enable: true
|
65
|
+
|
66
|
+
Sweatshop will attempt to connect to each server listed under
|
67
|
+
"cluster" in order, until it either manages to establish a connection
|
68
|
+
or until it runs out of servers.
|
69
|
+
|
70
|
+
If you only have a single Rabbit server, you can omit the "cluster"
|
71
|
+
option and just add "host" and "port" (or host: localhost:5672) options, as shown below:
|
72
|
+
|
73
|
+
default:
|
74
|
+
queue: rabbit
|
75
|
+
host: localhost
|
76
|
+
port: 5672
|
77
|
+
user: 'guest'
|
78
|
+
pass: 'guest'
|
79
|
+
vhost: '/'
|
80
|
+
enable: true
|
81
|
+
|
82
|
+
|
83
|
+
## Running the workers
|
84
|
+
|
85
|
+
Assuming you ran `rake setup` in Rails, you can type:
|
86
|
+
|
87
|
+
script/sweatshop
|
88
|
+
|
89
|
+
By default, the script will run all workers defined in the app/workers dir. Every task will be processed on each queue using a round-robin algorithm. You can also add the `-d` flag which will put the worker in daemon mode. The daemon also takes other params. Add a `-h` for more details.
|
90
|
+
|
91
|
+
script/sweatshop -d
|
92
|
+
script/sweatshop -d stop
|
93
|
+
|
94
|
+
If you would like to run SweatShop as a daemon on a linux machine, use the initd.sh script provided in the sweat_shop/script dir.
|
95
|
+
|
96
|
+
# REQUIREMENTS
|
97
|
+
|
98
|
+
memcache (for kestrel)
|
99
|
+
carrot (for rabbit)
|
100
|
+
|
101
|
+
# LICENSE
|
102
|
+
|
103
|
+
Copyright (c) 2009 Amos Elliston, Geni.com; Published under The MIT License, see License
|
data/Rakefile
ADDED
@@ -0,0 +1,43 @@
|
|
1
|
+
require 'rake'
|
2
|
+
require 'rake/testtask'
|
3
|
+
require 'rake/rdoctask'
|
4
|
+
require 'rcov/rcovtask'
|
5
|
+
|
6
|
+
begin
|
7
|
+
require 'jeweler'
|
8
|
+
Jeweler::Tasks.new do |s|
|
9
|
+
s.name = "sweatshop"
|
10
|
+
s.summary = %Q{Sweatshop is a simple asynchronous worker queue build on top of rabbitmq/ampq}
|
11
|
+
s.email = "amos@geni.com"
|
12
|
+
s.homepage = "http://github.com/famoseagle/sweat-shop"
|
13
|
+
s.description = "See summary"
|
14
|
+
s.authors = ["Amos Elliston"]
|
15
|
+
s.files = FileList["[A-Z]*", "install.rb", "{lib,test,config,script}/**/*"]
|
16
|
+
s.add_dependency('famoseagle-carrot', '= 0.7.0')
|
17
|
+
end
|
18
|
+
Jeweler::GemcutterTasks.new
|
19
|
+
rescue LoadError
|
20
|
+
puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
|
21
|
+
end
|
22
|
+
|
23
|
+
Rake::TestTask.new
|
24
|
+
|
25
|
+
Rake::RDocTask.new do |rdoc|
|
26
|
+
rdoc.rdoc_dir = 'rdoc'
|
27
|
+
rdoc.title = 'new_project'
|
28
|
+
rdoc.options << '--line-numbers' << '--inline-source'
|
29
|
+
rdoc.rdoc_files.include('README*')
|
30
|
+
rdoc.rdoc_files.include('lib/**/*.rb')
|
31
|
+
end
|
32
|
+
|
33
|
+
Rcov::RcovTask.new do |t|
|
34
|
+
t.libs << 'test'
|
35
|
+
t.test_files = FileList['test/**/*_test.rb']
|
36
|
+
t.verbose = true
|
37
|
+
end
|
38
|
+
|
39
|
+
task :default => :test
|
40
|
+
|
41
|
+
task :setup do
|
42
|
+
require File.dirname(__FILE__) + '/install'
|
43
|
+
end
|
data/VERSION.yml
ADDED
data/bin/sweatd
ADDED
data/config/defaults.yml
ADDED
data/install.rb
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
require 'fileutils'
|
2
|
+
root = File.dirname(__FILE__)
|
3
|
+
|
4
|
+
unless defined?(RAILS_ROOT)
|
5
|
+
search_paths = %W{/.. /../.. /../../..}
|
6
|
+
search_paths.each do |path|
|
7
|
+
if File.exist?(root + path + '/config/environment.rb')
|
8
|
+
RAILS_ROOT = root + path
|
9
|
+
break
|
10
|
+
end
|
11
|
+
end
|
12
|
+
end
|
13
|
+
return unless RAILS_ROOT
|
14
|
+
|
15
|
+
config = 'sweatshop.yml'
|
16
|
+
script = 'sweatshop'
|
17
|
+
|
18
|
+
FileUtils.cp(File.join(root, 'config', config), File.join(RAILS_ROOT, 'config', config))
|
19
|
+
FileUtils.cp(File.join(root, 'script', script), File.join(RAILS_ROOT, 'script', script))
|
20
|
+
puts "\n\ninstalled #{ [config, script].join(", ") } \n\n"
|
@@ -0,0 +1,17 @@
|
|
1
|
+
module MessageQueue
|
2
|
+
class Base
|
3
|
+
attr_reader :opts
|
4
|
+
def queue_size(queue); end
|
5
|
+
def enqueue(queue, data); end
|
6
|
+
def dequeue(queue); end
|
7
|
+
def confirm(queue); end
|
8
|
+
def subscribe(queue); end
|
9
|
+
def delete(queue); end
|
10
|
+
def client; end
|
11
|
+
def stop; end
|
12
|
+
|
13
|
+
def subscribe?
|
14
|
+
false
|
15
|
+
end
|
16
|
+
end
|
17
|
+
end
|
@@ -0,0 +1,32 @@
|
|
1
|
+
module MessageQueue
|
2
|
+
class Kestrel < Base
|
3
|
+
def initialize(opts)
|
4
|
+
@servers = opts['servers']
|
5
|
+
end
|
6
|
+
|
7
|
+
def queue_size(queue)
|
8
|
+
size = 0
|
9
|
+
stats = client.stats
|
10
|
+
servers.each do |server|
|
11
|
+
size += stats[server]["queue_#{queue}_items"].to_i
|
12
|
+
end
|
13
|
+
size
|
14
|
+
end
|
15
|
+
|
16
|
+
def enqueue(queue, data)
|
17
|
+
client.set(queue, data)
|
18
|
+
end
|
19
|
+
|
20
|
+
def dequeue(queue)
|
21
|
+
client.get("#{queue}/open")
|
22
|
+
end
|
23
|
+
|
24
|
+
def confirm(queue)
|
25
|
+
client.get("#{queue}/close")
|
26
|
+
end
|
27
|
+
|
28
|
+
def client
|
29
|
+
@client ||= MemCache.new(servers)
|
30
|
+
end
|
31
|
+
end
|
32
|
+
end
|
@@ -0,0 +1,108 @@
|
|
1
|
+
require 'carrot'
|
2
|
+
module MessageQueue
|
3
|
+
class Rabbit < Base
|
4
|
+
|
5
|
+
def initialize(opts={})
|
6
|
+
@opts = opts
|
7
|
+
end
|
8
|
+
|
9
|
+
def delete(queue)
|
10
|
+
send_command do
|
11
|
+
client.queue(queue).delete
|
12
|
+
end
|
13
|
+
end
|
14
|
+
|
15
|
+
def queue_size(queue)
|
16
|
+
send_command do
|
17
|
+
client.queue(queue).message_count
|
18
|
+
end
|
19
|
+
end
|
20
|
+
|
21
|
+
def enqueue(queue, data)
|
22
|
+
send_command do
|
23
|
+
client.queue(queue, :durable => true).publish(Marshal.dump(data), :persistent => true)
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
def dequeue(queue)
|
28
|
+
send_command do
|
29
|
+
task = client.queue(queue).pop(:ack => true)
|
30
|
+
return unless task
|
31
|
+
Marshal.load(task)
|
32
|
+
end
|
33
|
+
end
|
34
|
+
|
35
|
+
def confirm(queue)
|
36
|
+
send_command do
|
37
|
+
client.queue(queue).ack
|
38
|
+
end
|
39
|
+
end
|
40
|
+
|
41
|
+
def send_command(&block)
|
42
|
+
retried = false
|
43
|
+
begin
|
44
|
+
block.call
|
45
|
+
rescue Carrot::AMQP::Server::ServerDown => e
|
46
|
+
if not retried
|
47
|
+
puts "Error #{e.message}. Retrying..."
|
48
|
+
@client = nil
|
49
|
+
retried = true
|
50
|
+
retry
|
51
|
+
else
|
52
|
+
raise e
|
53
|
+
end
|
54
|
+
end
|
55
|
+
end
|
56
|
+
|
57
|
+
def client
|
58
|
+
return @client if @client
|
59
|
+
|
60
|
+
if @opts['cluster']
|
61
|
+
@opts['cluster'].each_with_index do |server, i|
|
62
|
+
host, port = server.split(':')
|
63
|
+
begin
|
64
|
+
@client = Carrot.new(
|
65
|
+
:host => host,
|
66
|
+
:port => port.to_i,
|
67
|
+
:user => @opts['user'],
|
68
|
+
:pass => @opts['pass'],
|
69
|
+
:vhost => @opts['vhost'],
|
70
|
+
:insist => @opts['insist']
|
71
|
+
)
|
72
|
+
return @client
|
73
|
+
rescue Carrot::AMQP::Server::ServerDown => e
|
74
|
+
if i == (@opts['cluster'].size-1)
|
75
|
+
raise e
|
76
|
+
else
|
77
|
+
next
|
78
|
+
end
|
79
|
+
end
|
80
|
+
end
|
81
|
+
else
|
82
|
+
if @opts['host'] =~ /:/
|
83
|
+
host, port = @opts['host'].split(':')
|
84
|
+
else
|
85
|
+
host = @opts['host']
|
86
|
+
port = @opts['port']
|
87
|
+
end
|
88
|
+
@client = Carrot.new(
|
89
|
+
:host => host,
|
90
|
+
:port => port.to_i,
|
91
|
+
:user => @opts['user'],
|
92
|
+
:pass => @opts['pass'],
|
93
|
+
:vhost => @opts['vhost'],
|
94
|
+
:insist => @opts['insist']
|
95
|
+
)
|
96
|
+
end
|
97
|
+
@client
|
98
|
+
end
|
99
|
+
|
100
|
+
def client=(client)
|
101
|
+
@client = client
|
102
|
+
end
|
103
|
+
|
104
|
+
def stop
|
105
|
+
client.stop
|
106
|
+
end
|
107
|
+
end
|
108
|
+
end
|
data/lib/sweat_shop.rb
ADDED
@@ -0,0 +1,179 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'digest'
|
3
|
+
require 'yaml'
|
4
|
+
|
5
|
+
$:.unshift(File.dirname(__FILE__))
|
6
|
+
require 'message_queue/base'
|
7
|
+
require 'message_queue/rabbit'
|
8
|
+
require 'message_queue/kestrel'
|
9
|
+
require 'sweat_shop/worker'
|
10
|
+
|
11
|
+
module SweatShop
|
12
|
+
extend self
|
13
|
+
|
14
|
+
def workers
|
15
|
+
@workers ||= []
|
16
|
+
end
|
17
|
+
|
18
|
+
def workers=(workers)
|
19
|
+
@workers = workers
|
20
|
+
end
|
21
|
+
|
22
|
+
def workers_in_group(groups)
|
23
|
+
groups = [groups] unless groups.is_a?(Array)
|
24
|
+
if groups.include?(:all)
|
25
|
+
workers
|
26
|
+
else
|
27
|
+
workers.select do |worker|
|
28
|
+
groups.include?(worker.queue_group)
|
29
|
+
end
|
30
|
+
end
|
31
|
+
end
|
32
|
+
|
33
|
+
def do_tasks(workers)
|
34
|
+
if queue.subscribe?
|
35
|
+
EM.run do
|
36
|
+
workers.each do |worker|
|
37
|
+
worker.subscribe
|
38
|
+
end
|
39
|
+
end
|
40
|
+
else
|
41
|
+
loop do
|
42
|
+
wait = true
|
43
|
+
workers.each do |worker|
|
44
|
+
if task = worker.dequeue
|
45
|
+
worker.do_task(task)
|
46
|
+
wait = false
|
47
|
+
end
|
48
|
+
end
|
49
|
+
if stop?
|
50
|
+
workers.each do |worker|
|
51
|
+
worker.stop
|
52
|
+
end
|
53
|
+
queue.stop
|
54
|
+
exit
|
55
|
+
end
|
56
|
+
sleep 1 if wait
|
57
|
+
end
|
58
|
+
end
|
59
|
+
end
|
60
|
+
|
61
|
+
def do_all_tasks
|
62
|
+
do_tasks(
|
63
|
+
workers_in_group(:all)
|
64
|
+
)
|
65
|
+
end
|
66
|
+
|
67
|
+
def do_default_tasks
|
68
|
+
do_tasks(
|
69
|
+
workers_in_group(:default)
|
70
|
+
)
|
71
|
+
end
|
72
|
+
|
73
|
+
def config
|
74
|
+
@config ||= begin
|
75
|
+
defaults = YAML.load_file(File.dirname(__FILE__) + '/../config/defaults.yml')
|
76
|
+
if defined?(RAILS_ROOT)
|
77
|
+
file = RAILS_ROOT + '/config/sweatshop.yml'
|
78
|
+
if File.exist?(file)
|
79
|
+
YAML.load_file(file)[RAILS_ENV || 'development']
|
80
|
+
else
|
81
|
+
defaults['enable'] = false
|
82
|
+
defaults
|
83
|
+
end
|
84
|
+
else
|
85
|
+
defaults
|
86
|
+
end
|
87
|
+
end
|
88
|
+
end
|
89
|
+
|
90
|
+
def stop
|
91
|
+
@stop = true
|
92
|
+
queue.stop if queue.subscribe?
|
93
|
+
end
|
94
|
+
|
95
|
+
def stop?
|
96
|
+
@stop
|
97
|
+
end
|
98
|
+
|
99
|
+
def queue_sizes
|
100
|
+
workers.inject([]) do |all, worker|
|
101
|
+
all << [worker, worker.queue_size]
|
102
|
+
all
|
103
|
+
end
|
104
|
+
end
|
105
|
+
|
106
|
+
def queue(type = 'default')
|
107
|
+
type = config[type] ? type : 'default'
|
108
|
+
return queues[type] if queues[type]
|
109
|
+
|
110
|
+
qconfig = config[type]
|
111
|
+
qtype = qconfig['queue'] || 'rabbit'
|
112
|
+
queue = constantize("MessageQueue::#{qtype.capitalize}")
|
113
|
+
|
114
|
+
queues[type] = queue.new(qconfig)
|
115
|
+
end
|
116
|
+
|
117
|
+
def queue=(queue, type = 'default')
|
118
|
+
queues[type] = queue
|
119
|
+
end
|
120
|
+
|
121
|
+
def queues
|
122
|
+
@queues ||= {}
|
123
|
+
end
|
124
|
+
|
125
|
+
def queue_groups
|
126
|
+
@queue_groups ||= workers.collect{|w| w.queue_group} << 'default'
|
127
|
+
end
|
128
|
+
|
129
|
+
def pp_sizes
|
130
|
+
max_width = workers.collect{|w| w.to_s.size}.max
|
131
|
+
puts '-' * (max_width + 10)
|
132
|
+
puts queue_sizes.collect{ |p| sprintf("%-#{max_width}s %2s", p.first, p.last) }.join("\n")
|
133
|
+
puts '-' * (max_width + 10)
|
134
|
+
end
|
135
|
+
|
136
|
+
def cluster_info
|
137
|
+
servers = []
|
138
|
+
queue_groups.each do |group|
|
139
|
+
qconfig = config[group]
|
140
|
+
next unless qconfig
|
141
|
+
next unless qconfig['cluster']
|
142
|
+
servers << qconfig['cluster']
|
143
|
+
end
|
144
|
+
servers.flatten!
|
145
|
+
|
146
|
+
servers.each do |server|
|
147
|
+
puts "\nQueue sizes on #{server}"
|
148
|
+
queue = MessageQueue::Rabbit.new('host' => server)
|
149
|
+
queue_groups.each do |group|
|
150
|
+
queues[group] = queue
|
151
|
+
end
|
152
|
+
pp_sizes
|
153
|
+
puts
|
154
|
+
end
|
155
|
+
@queues = {}
|
156
|
+
nil
|
157
|
+
end
|
158
|
+
|
159
|
+
def log(msg)
|
160
|
+
return if logger == :silent
|
161
|
+
logger ? logger.debug(msg) : puts(msg)
|
162
|
+
end
|
163
|
+
|
164
|
+
def logger
|
165
|
+
@logger
|
166
|
+
end
|
167
|
+
|
168
|
+
def logger=(logger)
|
169
|
+
@logger = logger
|
170
|
+
end
|
171
|
+
|
172
|
+
def constantize(str)
|
173
|
+
Object.module_eval("#{str}", __FILE__, __LINE__)
|
174
|
+
end
|
175
|
+
end
|
176
|
+
|
177
|
+
if defined?(RAILS_ROOT)
|
178
|
+
Dir.glob(RAILS_ROOT + '/app/workers/*.rb').each{|worker| require worker }
|
179
|
+
end
|