sweatshop 1.4.0

Sign up to get free protection for your applications and to get access to all the features.
data/History.txt ADDED
@@ -0,0 +1,6 @@
1
+ === 1.0.0 / 2009-01-13
2
+
3
+ * 1 major enhancement
4
+
5
+ * Birthday!
6
+
data/LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) 2009 Amos Elliston
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
data/README.markdown ADDED
@@ -0,0 +1,103 @@
1
+ # SweatShop
2
+
3
+ SweatShop provides an api to background resource intensive tasks. Much of the api design was copied from Workling, with a few tweaks.
4
+ Currently, it runs rabbitmq and kestrel, but it can support any number of queues.
5
+
6
+ ## Installing
7
+
8
+ gem install sweat_shop
9
+ freeze in your gems directory (add config.gem 'sweat_shop' to your environment)
10
+ cd vendor/gems/sweat_shop
11
+ rake setup
12
+
13
+ ## Writing workers
14
+
15
+ Put `email_worker.rb` into app/workers and sublcass `SweatShop::Worker`:
16
+
17
+ class EmailWorker < SweatShop::Worker
18
+ def send_mail(to)
19
+ user = User.find_by_id(to)
20
+ Mailer.deliver_welcome(to)
21
+ end
22
+ end
23
+
24
+ Then, anywhere in your app you can execute:
25
+
26
+ EmailWorker.async_send_mail(1)
27
+
28
+ The `async` signifies that this task will be placed on a queue to be serviced by the EmailWorker possibly on another machine. You can also
29
+ call:
30
+
31
+ EmailWorker.send_mail(1)
32
+
33
+ That will do the work immediately, without placing the task on the queue. You can also define a `queue_group` at the top of the file
34
+ which will allow you to split workers out into logical groups. This is important if you have various machines serving different
35
+ queues.
36
+
37
+ ## Running the queue
38
+
39
+ SweatShop has been tested with Rabbit and Kestrel, but it will also work with Starling. Please use the following resources to install the server:
40
+
41
+ Kestrel:
42
+ http://github.com/robey/kestrel/tree/master
43
+
44
+ Rabbit:
45
+ http://github.com/ezmobius/nanite/tree/master
46
+
47
+ config/sweatshop.yml specifies the machine address of the queue
48
+ (default localhost:5672). You can also specify the queue type with the
49
+ queue param.
50
+
51
+ ## Rabbit cluster support
52
+
53
+ The following example configuration shows support for Rabbit clusters
54
+ within a queue group:
55
+
56
+ default:
57
+ queue: rabbit
58
+ cluster:
59
+ - hostA:5672
60
+ - hostB:5672
61
+ user: 'guest'
62
+ pass: 'guest'
63
+ vhost: '/'
64
+ enable: true
65
+
66
+ Sweatshop will attempt to connect to each server listed under
67
+ "cluster" in order, until it either manages to establish a connection
68
+ or until it runs out of servers.
69
+
70
+ If you only have a single Rabbit server, you can omit the "cluster"
71
+ option and just add "host" and "port" (or host: localhost:5672) options, as shown below:
72
+
73
+ default:
74
+ queue: rabbit
75
+ host: localhost
76
+ port: 5672
77
+ user: 'guest'
78
+ pass: 'guest'
79
+ vhost: '/'
80
+ enable: true
81
+
82
+
83
+ ## Running the workers
84
+
85
+ Assuming you ran `rake setup` in Rails, you can type:
86
+
87
+ script/sweatshop
88
+
89
+ By default, the script will run all workers defined in the app/workers dir. Every task will be processed on each queue using a round-robin algorithm. You can also add the `-d` flag which will put the worker in daemon mode. The daemon also takes other params. Add a `-h` for more details.
90
+
91
+ script/sweatshop -d
92
+ script/sweatshop -d stop
93
+
94
+ If you would like to run SweatShop as a daemon on a linux machine, use the initd.sh script provided in the sweat_shop/script dir.
95
+
96
+ # REQUIREMENTS
97
+
98
+ memcache (for kestrel)
99
+ carrot (for rabbit)
100
+
101
+ # LICENSE
102
+
103
+ Copyright (c) 2009 Amos Elliston, Geni.com; Published under The MIT License, see License
data/Rakefile ADDED
@@ -0,0 +1,43 @@
1
+ require 'rake'
2
+ require 'rake/testtask'
3
+ require 'rake/rdoctask'
4
+ require 'rcov/rcovtask'
5
+
6
+ begin
7
+ require 'jeweler'
8
+ Jeweler::Tasks.new do |s|
9
+ s.name = "sweatshop"
10
+ s.summary = %Q{Sweatshop is a simple asynchronous worker queue build on top of rabbitmq/ampq}
11
+ s.email = "amos@geni.com"
12
+ s.homepage = "http://github.com/famoseagle/sweat-shop"
13
+ s.description = "See summary"
14
+ s.authors = ["Amos Elliston"]
15
+ s.files = FileList["[A-Z]*", "install.rb", "{lib,test,config,script}/**/*"]
16
+ s.add_dependency('famoseagle-carrot', '= 0.7.0')
17
+ end
18
+ Jeweler::GemcutterTasks.new
19
+ rescue LoadError
20
+ puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
21
+ end
22
+
23
+ Rake::TestTask.new
24
+
25
+ Rake::RDocTask.new do |rdoc|
26
+ rdoc.rdoc_dir = 'rdoc'
27
+ rdoc.title = 'new_project'
28
+ rdoc.options << '--line-numbers' << '--inline-source'
29
+ rdoc.rdoc_files.include('README*')
30
+ rdoc.rdoc_files.include('lib/**/*.rb')
31
+ end
32
+
33
+ Rcov::RcovTask.new do |t|
34
+ t.libs << 'test'
35
+ t.test_files = FileList['test/**/*_test.rb']
36
+ t.verbose = true
37
+ end
38
+
39
+ task :default => :test
40
+
41
+ task :setup do
42
+ require File.dirname(__FILE__) + '/install'
43
+ end
data/VERSION.yml ADDED
@@ -0,0 +1,4 @@
1
+ ---
2
+ :patch: 0
3
+ :major: 1
4
+ :minor: 4
data/bin/sweatd ADDED
@@ -0,0 +1,3 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require File.dirname(__FILE__) + '/../lib/sweat_shop/sweatd'
@@ -0,0 +1,8 @@
1
+ # default options
2
+ default:
3
+ queue: rabbit
4
+ host: localhost:5672
5
+ user: 'guest'
6
+ pass: 'guest'
7
+ vhost: '/'
8
+ enable: true
@@ -0,0 +1,15 @@
1
+ development:
2
+ default:
3
+ queue: rabbit
4
+ host: localhost:5672
5
+ enable: true
6
+ test:
7
+ default:
8
+ queue: rabbit
9
+ host: localhost:5672
10
+ enable: true
11
+ production:
12
+ default:
13
+ queue: rabbit
14
+ host: localhost:5672
15
+ enable: true
data/install.rb ADDED
@@ -0,0 +1,20 @@
1
+ require 'fileutils'
2
+ root = File.dirname(__FILE__)
3
+
4
+ unless defined?(RAILS_ROOT)
5
+ search_paths = %W{/.. /../.. /../../..}
6
+ search_paths.each do |path|
7
+ if File.exist?(root + path + '/config/environment.rb')
8
+ RAILS_ROOT = root + path
9
+ break
10
+ end
11
+ end
12
+ end
13
+ return unless RAILS_ROOT
14
+
15
+ config = 'sweatshop.yml'
16
+ script = 'sweatshop'
17
+
18
+ FileUtils.cp(File.join(root, 'config', config), File.join(RAILS_ROOT, 'config', config))
19
+ FileUtils.cp(File.join(root, 'script', script), File.join(RAILS_ROOT, 'script', script))
20
+ puts "\n\ninstalled #{ [config, script].join(", ") } \n\n"
@@ -0,0 +1,17 @@
1
+ module MessageQueue
2
+ class Base
3
+ attr_reader :opts
4
+ def queue_size(queue); end
5
+ def enqueue(queue, data); end
6
+ def dequeue(queue); end
7
+ def confirm(queue); end
8
+ def subscribe(queue); end
9
+ def delete(queue); end
10
+ def client; end
11
+ def stop; end
12
+
13
+ def subscribe?
14
+ false
15
+ end
16
+ end
17
+ end
@@ -0,0 +1,32 @@
1
+ module MessageQueue
2
+ class Kestrel < Base
3
+ def initialize(opts)
4
+ @servers = opts['servers']
5
+ end
6
+
7
+ def queue_size(queue)
8
+ size = 0
9
+ stats = client.stats
10
+ servers.each do |server|
11
+ size += stats[server]["queue_#{queue}_items"].to_i
12
+ end
13
+ size
14
+ end
15
+
16
+ def enqueue(queue, data)
17
+ client.set(queue, data)
18
+ end
19
+
20
+ def dequeue(queue)
21
+ client.get("#{queue}/open")
22
+ end
23
+
24
+ def confirm(queue)
25
+ client.get("#{queue}/close")
26
+ end
27
+
28
+ def client
29
+ @client ||= MemCache.new(servers)
30
+ end
31
+ end
32
+ end
@@ -0,0 +1,108 @@
1
+ require 'carrot'
2
+ module MessageQueue
3
+ class Rabbit < Base
4
+
5
+ def initialize(opts={})
6
+ @opts = opts
7
+ end
8
+
9
+ def delete(queue)
10
+ send_command do
11
+ client.queue(queue).delete
12
+ end
13
+ end
14
+
15
+ def queue_size(queue)
16
+ send_command do
17
+ client.queue(queue).message_count
18
+ end
19
+ end
20
+
21
+ def enqueue(queue, data)
22
+ send_command do
23
+ client.queue(queue, :durable => true).publish(Marshal.dump(data), :persistent => true)
24
+ end
25
+ end
26
+
27
+ def dequeue(queue)
28
+ send_command do
29
+ task = client.queue(queue).pop(:ack => true)
30
+ return unless task
31
+ Marshal.load(task)
32
+ end
33
+ end
34
+
35
+ def confirm(queue)
36
+ send_command do
37
+ client.queue(queue).ack
38
+ end
39
+ end
40
+
41
+ def send_command(&block)
42
+ retried = false
43
+ begin
44
+ block.call
45
+ rescue Carrot::AMQP::Server::ServerDown => e
46
+ if not retried
47
+ puts "Error #{e.message}. Retrying..."
48
+ @client = nil
49
+ retried = true
50
+ retry
51
+ else
52
+ raise e
53
+ end
54
+ end
55
+ end
56
+
57
+ def client
58
+ return @client if @client
59
+
60
+ if @opts['cluster']
61
+ @opts['cluster'].each_with_index do |server, i|
62
+ host, port = server.split(':')
63
+ begin
64
+ @client = Carrot.new(
65
+ :host => host,
66
+ :port => port.to_i,
67
+ :user => @opts['user'],
68
+ :pass => @opts['pass'],
69
+ :vhost => @opts['vhost'],
70
+ :insist => @opts['insist']
71
+ )
72
+ return @client
73
+ rescue Carrot::AMQP::Server::ServerDown => e
74
+ if i == (@opts['cluster'].size-1)
75
+ raise e
76
+ else
77
+ next
78
+ end
79
+ end
80
+ end
81
+ else
82
+ if @opts['host'] =~ /:/
83
+ host, port = @opts['host'].split(':')
84
+ else
85
+ host = @opts['host']
86
+ port = @opts['port']
87
+ end
88
+ @client = Carrot.new(
89
+ :host => host,
90
+ :port => port.to_i,
91
+ :user => @opts['user'],
92
+ :pass => @opts['pass'],
93
+ :vhost => @opts['vhost'],
94
+ :insist => @opts['insist']
95
+ )
96
+ end
97
+ @client
98
+ end
99
+
100
+ def client=(client)
101
+ @client = client
102
+ end
103
+
104
+ def stop
105
+ client.stop
106
+ end
107
+ end
108
+ end
data/lib/sweat_shop.rb ADDED
@@ -0,0 +1,179 @@
1
+ require 'rubygems'
2
+ require 'digest'
3
+ require 'yaml'
4
+
5
+ $:.unshift(File.dirname(__FILE__))
6
+ require 'message_queue/base'
7
+ require 'message_queue/rabbit'
8
+ require 'message_queue/kestrel'
9
+ require 'sweat_shop/worker'
10
+
11
+ module SweatShop
12
+ extend self
13
+
14
+ def workers
15
+ @workers ||= []
16
+ end
17
+
18
+ def workers=(workers)
19
+ @workers = workers
20
+ end
21
+
22
+ def workers_in_group(groups)
23
+ groups = [groups] unless groups.is_a?(Array)
24
+ if groups.include?(:all)
25
+ workers
26
+ else
27
+ workers.select do |worker|
28
+ groups.include?(worker.queue_group)
29
+ end
30
+ end
31
+ end
32
+
33
+ def do_tasks(workers)
34
+ if queue.subscribe?
35
+ EM.run do
36
+ workers.each do |worker|
37
+ worker.subscribe
38
+ end
39
+ end
40
+ else
41
+ loop do
42
+ wait = true
43
+ workers.each do |worker|
44
+ if task = worker.dequeue
45
+ worker.do_task(task)
46
+ wait = false
47
+ end
48
+ end
49
+ if stop?
50
+ workers.each do |worker|
51
+ worker.stop
52
+ end
53
+ queue.stop
54
+ exit
55
+ end
56
+ sleep 1 if wait
57
+ end
58
+ end
59
+ end
60
+
61
+ def do_all_tasks
62
+ do_tasks(
63
+ workers_in_group(:all)
64
+ )
65
+ end
66
+
67
+ def do_default_tasks
68
+ do_tasks(
69
+ workers_in_group(:default)
70
+ )
71
+ end
72
+
73
+ def config
74
+ @config ||= begin
75
+ defaults = YAML.load_file(File.dirname(__FILE__) + '/../config/defaults.yml')
76
+ if defined?(RAILS_ROOT)
77
+ file = RAILS_ROOT + '/config/sweatshop.yml'
78
+ if File.exist?(file)
79
+ YAML.load_file(file)[RAILS_ENV || 'development']
80
+ else
81
+ defaults['enable'] = false
82
+ defaults
83
+ end
84
+ else
85
+ defaults
86
+ end
87
+ end
88
+ end
89
+
90
+ def stop
91
+ @stop = true
92
+ queue.stop if queue.subscribe?
93
+ end
94
+
95
+ def stop?
96
+ @stop
97
+ end
98
+
99
+ def queue_sizes
100
+ workers.inject([]) do |all, worker|
101
+ all << [worker, worker.queue_size]
102
+ all
103
+ end
104
+ end
105
+
106
+ def queue(type = 'default')
107
+ type = config[type] ? type : 'default'
108
+ return queues[type] if queues[type]
109
+
110
+ qconfig = config[type]
111
+ qtype = qconfig['queue'] || 'rabbit'
112
+ queue = constantize("MessageQueue::#{qtype.capitalize}")
113
+
114
+ queues[type] = queue.new(qconfig)
115
+ end
116
+
117
+ def queue=(queue, type = 'default')
118
+ queues[type] = queue
119
+ end
120
+
121
+ def queues
122
+ @queues ||= {}
123
+ end
124
+
125
+ def queue_groups
126
+ @queue_groups ||= workers.collect{|w| w.queue_group} << 'default'
127
+ end
128
+
129
+ def pp_sizes
130
+ max_width = workers.collect{|w| w.to_s.size}.max
131
+ puts '-' * (max_width + 10)
132
+ puts queue_sizes.collect{ |p| sprintf("%-#{max_width}s %2s", p.first, p.last) }.join("\n")
133
+ puts '-' * (max_width + 10)
134
+ end
135
+
136
+ def cluster_info
137
+ servers = []
138
+ queue_groups.each do |group|
139
+ qconfig = config[group]
140
+ next unless qconfig
141
+ next unless qconfig['cluster']
142
+ servers << qconfig['cluster']
143
+ end
144
+ servers.flatten!
145
+
146
+ servers.each do |server|
147
+ puts "\nQueue sizes on #{server}"
148
+ queue = MessageQueue::Rabbit.new('host' => server)
149
+ queue_groups.each do |group|
150
+ queues[group] = queue
151
+ end
152
+ pp_sizes
153
+ puts
154
+ end
155
+ @queues = {}
156
+ nil
157
+ end
158
+
159
+ def log(msg)
160
+ return if logger == :silent
161
+ logger ? logger.debug(msg) : puts(msg)
162
+ end
163
+
164
+ def logger
165
+ @logger
166
+ end
167
+
168
+ def logger=(logger)
169
+ @logger = logger
170
+ end
171
+
172
+ def constantize(str)
173
+ Object.module_eval("#{str}", __FILE__, __LINE__)
174
+ end
175
+ end
176
+
177
+ if defined?(RAILS_ROOT)
178
+ Dir.glob(RAILS_ROOT + '/app/workers/*.rb').each{|worker| require worker }
179
+ end