sweatshop 1.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/History.txt ADDED
@@ -0,0 +1,6 @@
1
+ === 1.0.0 / 2009-01-13
2
+
3
+ * 1 major enhancement
4
+
5
+ * Birthday!
6
+
data/LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) 2009 Amos Elliston
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
data/README.markdown ADDED
@@ -0,0 +1,103 @@
1
+ # SweatShop
2
+
3
+ SweatShop provides an api to background resource intensive tasks. Much of the api design was copied from Workling, with a few tweaks.
4
+ Currently, it runs rabbitmq and kestrel, but it can support any number of queues.
5
+
6
+ ## Installing
7
+
8
+ gem install sweat_shop
9
+ freeze in your gems directory (add config.gem 'sweat_shop' to your environment)
10
+ cd vendor/gems/sweat_shop
11
+ rake setup
12
+
13
+ ## Writing workers
14
+
15
+ Put `email_worker.rb` into app/workers and sublcass `SweatShop::Worker`:
16
+
17
+ class EmailWorker < SweatShop::Worker
18
+ def send_mail(to)
19
+ user = User.find_by_id(to)
20
+ Mailer.deliver_welcome(to)
21
+ end
22
+ end
23
+
24
+ Then, anywhere in your app you can execute:
25
+
26
+ EmailWorker.async_send_mail(1)
27
+
28
+ The `async` signifies that this task will be placed on a queue to be serviced by the EmailWorker possibly on another machine. You can also
29
+ call:
30
+
31
+ EmailWorker.send_mail(1)
32
+
33
+ That will do the work immediately, without placing the task on the queue. You can also define a `queue_group` at the top of the file
34
+ which will allow you to split workers out into logical groups. This is important if you have various machines serving different
35
+ queues.
36
+
37
+ ## Running the queue
38
+
39
+ SweatShop has been tested with Rabbit and Kestrel, but it will also work with Starling. Please use the following resources to install the server:
40
+
41
+ Kestrel:
42
+ http://github.com/robey/kestrel/tree/master
43
+
44
+ Rabbit:
45
+ http://github.com/ezmobius/nanite/tree/master
46
+
47
+ config/sweatshop.yml specifies the machine address of the queue
48
+ (default localhost:5672). You can also specify the queue type with the
49
+ queue param.
50
+
51
+ ## Rabbit cluster support
52
+
53
+ The following example configuration shows support for Rabbit clusters
54
+ within a queue group:
55
+
56
+ default:
57
+ queue: rabbit
58
+ cluster:
59
+ - hostA:5672
60
+ - hostB:5672
61
+ user: 'guest'
62
+ pass: 'guest'
63
+ vhost: '/'
64
+ enable: true
65
+
66
+ Sweatshop will attempt to connect to each server listed under
67
+ "cluster" in order, until it either manages to establish a connection
68
+ or until it runs out of servers.
69
+
70
+ If you only have a single Rabbit server, you can omit the "cluster"
71
+ option and just add "host" and "port" (or host: localhost:5672) options, as shown below:
72
+
73
+ default:
74
+ queue: rabbit
75
+ host: localhost
76
+ port: 5672
77
+ user: 'guest'
78
+ pass: 'guest'
79
+ vhost: '/'
80
+ enable: true
81
+
82
+
83
+ ## Running the workers
84
+
85
+ Assuming you ran `rake setup` in Rails, you can type:
86
+
87
+ script/sweatshop
88
+
89
+ By default, the script will run all workers defined in the app/workers dir. Every task will be processed on each queue using a round-robin algorithm. You can also add the `-d` flag which will put the worker in daemon mode. The daemon also takes other params. Add a `-h` for more details.
90
+
91
+ script/sweatshop -d
92
+ script/sweatshop -d stop
93
+
94
+ If you would like to run SweatShop as a daemon on a linux machine, use the initd.sh script provided in the sweat_shop/script dir.
95
+
96
+ # REQUIREMENTS
97
+
98
+ memcache (for kestrel)
99
+ carrot (for rabbit)
100
+
101
+ # LICENSE
102
+
103
+ Copyright (c) 2009 Amos Elliston, Geni.com; Published under The MIT License, see License
data/Rakefile ADDED
@@ -0,0 +1,43 @@
1
+ require 'rake'
2
+ require 'rake/testtask'
3
+ require 'rake/rdoctask'
4
+ require 'rcov/rcovtask'
5
+
6
+ begin
7
+ require 'jeweler'
8
+ Jeweler::Tasks.new do |s|
9
+ s.name = "sweatshop"
10
+ s.summary = %Q{Sweatshop is a simple asynchronous worker queue build on top of rabbitmq/ampq}
11
+ s.email = "amos@geni.com"
12
+ s.homepage = "http://github.com/famoseagle/sweat-shop"
13
+ s.description = "See summary"
14
+ s.authors = ["Amos Elliston"]
15
+ s.files = FileList["[A-Z]*", "install.rb", "{lib,test,config,script}/**/*"]
16
+ s.add_dependency('famoseagle-carrot', '= 0.7.0')
17
+ end
18
+ Jeweler::GemcutterTasks.new
19
+ rescue LoadError
20
+ puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
21
+ end
22
+
23
+ Rake::TestTask.new
24
+
25
+ Rake::RDocTask.new do |rdoc|
26
+ rdoc.rdoc_dir = 'rdoc'
27
+ rdoc.title = 'new_project'
28
+ rdoc.options << '--line-numbers' << '--inline-source'
29
+ rdoc.rdoc_files.include('README*')
30
+ rdoc.rdoc_files.include('lib/**/*.rb')
31
+ end
32
+
33
+ Rcov::RcovTask.new do |t|
34
+ t.libs << 'test'
35
+ t.test_files = FileList['test/**/*_test.rb']
36
+ t.verbose = true
37
+ end
38
+
39
+ task :default => :test
40
+
41
+ task :setup do
42
+ require File.dirname(__FILE__) + '/install'
43
+ end
data/VERSION.yml ADDED
@@ -0,0 +1,4 @@
1
+ ---
2
+ :patch: 0
3
+ :major: 1
4
+ :minor: 4
data/bin/sweatd ADDED
@@ -0,0 +1,3 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require File.dirname(__FILE__) + '/../lib/sweat_shop/sweatd'
@@ -0,0 +1,8 @@
1
+ # default options
2
+ default:
3
+ queue: rabbit
4
+ host: localhost:5672
5
+ user: 'guest'
6
+ pass: 'guest'
7
+ vhost: '/'
8
+ enable: true
@@ -0,0 +1,15 @@
1
+ development:
2
+ default:
3
+ queue: rabbit
4
+ host: localhost:5672
5
+ enable: true
6
+ test:
7
+ default:
8
+ queue: rabbit
9
+ host: localhost:5672
10
+ enable: true
11
+ production:
12
+ default:
13
+ queue: rabbit
14
+ host: localhost:5672
15
+ enable: true
data/install.rb ADDED
@@ -0,0 +1,20 @@
1
+ require 'fileutils'
2
+ root = File.dirname(__FILE__)
3
+
4
+ unless defined?(RAILS_ROOT)
5
+ search_paths = %W{/.. /../.. /../../..}
6
+ search_paths.each do |path|
7
+ if File.exist?(root + path + '/config/environment.rb')
8
+ RAILS_ROOT = root + path
9
+ break
10
+ end
11
+ end
12
+ end
13
+ return unless RAILS_ROOT
14
+
15
+ config = 'sweatshop.yml'
16
+ script = 'sweatshop'
17
+
18
+ FileUtils.cp(File.join(root, 'config', config), File.join(RAILS_ROOT, 'config', config))
19
+ FileUtils.cp(File.join(root, 'script', script), File.join(RAILS_ROOT, 'script', script))
20
+ puts "\n\ninstalled #{ [config, script].join(", ") } \n\n"
@@ -0,0 +1,17 @@
1
+ module MessageQueue
2
+ class Base
3
+ attr_reader :opts
4
+ def queue_size(queue); end
5
+ def enqueue(queue, data); end
6
+ def dequeue(queue); end
7
+ def confirm(queue); end
8
+ def subscribe(queue); end
9
+ def delete(queue); end
10
+ def client; end
11
+ def stop; end
12
+
13
+ def subscribe?
14
+ false
15
+ end
16
+ end
17
+ end
@@ -0,0 +1,32 @@
1
+ module MessageQueue
2
+ class Kestrel < Base
3
+ def initialize(opts)
4
+ @servers = opts['servers']
5
+ end
6
+
7
+ def queue_size(queue)
8
+ size = 0
9
+ stats = client.stats
10
+ servers.each do |server|
11
+ size += stats[server]["queue_#{queue}_items"].to_i
12
+ end
13
+ size
14
+ end
15
+
16
+ def enqueue(queue, data)
17
+ client.set(queue, data)
18
+ end
19
+
20
+ def dequeue(queue)
21
+ client.get("#{queue}/open")
22
+ end
23
+
24
+ def confirm(queue)
25
+ client.get("#{queue}/close")
26
+ end
27
+
28
+ def client
29
+ @client ||= MemCache.new(servers)
30
+ end
31
+ end
32
+ end
@@ -0,0 +1,108 @@
1
+ require 'carrot'
2
+ module MessageQueue
3
+ class Rabbit < Base
4
+
5
+ def initialize(opts={})
6
+ @opts = opts
7
+ end
8
+
9
+ def delete(queue)
10
+ send_command do
11
+ client.queue(queue).delete
12
+ end
13
+ end
14
+
15
+ def queue_size(queue)
16
+ send_command do
17
+ client.queue(queue).message_count
18
+ end
19
+ end
20
+
21
+ def enqueue(queue, data)
22
+ send_command do
23
+ client.queue(queue, :durable => true).publish(Marshal.dump(data), :persistent => true)
24
+ end
25
+ end
26
+
27
+ def dequeue(queue)
28
+ send_command do
29
+ task = client.queue(queue).pop(:ack => true)
30
+ return unless task
31
+ Marshal.load(task)
32
+ end
33
+ end
34
+
35
+ def confirm(queue)
36
+ send_command do
37
+ client.queue(queue).ack
38
+ end
39
+ end
40
+
41
+ def send_command(&block)
42
+ retried = false
43
+ begin
44
+ block.call
45
+ rescue Carrot::AMQP::Server::ServerDown => e
46
+ if not retried
47
+ puts "Error #{e.message}. Retrying..."
48
+ @client = nil
49
+ retried = true
50
+ retry
51
+ else
52
+ raise e
53
+ end
54
+ end
55
+ end
56
+
57
+ def client
58
+ return @client if @client
59
+
60
+ if @opts['cluster']
61
+ @opts['cluster'].each_with_index do |server, i|
62
+ host, port = server.split(':')
63
+ begin
64
+ @client = Carrot.new(
65
+ :host => host,
66
+ :port => port.to_i,
67
+ :user => @opts['user'],
68
+ :pass => @opts['pass'],
69
+ :vhost => @opts['vhost'],
70
+ :insist => @opts['insist']
71
+ )
72
+ return @client
73
+ rescue Carrot::AMQP::Server::ServerDown => e
74
+ if i == (@opts['cluster'].size-1)
75
+ raise e
76
+ else
77
+ next
78
+ end
79
+ end
80
+ end
81
+ else
82
+ if @opts['host'] =~ /:/
83
+ host, port = @opts['host'].split(':')
84
+ else
85
+ host = @opts['host']
86
+ port = @opts['port']
87
+ end
88
+ @client = Carrot.new(
89
+ :host => host,
90
+ :port => port.to_i,
91
+ :user => @opts['user'],
92
+ :pass => @opts['pass'],
93
+ :vhost => @opts['vhost'],
94
+ :insist => @opts['insist']
95
+ )
96
+ end
97
+ @client
98
+ end
99
+
100
+ def client=(client)
101
+ @client = client
102
+ end
103
+
104
+ def stop
105
+ client.stop
106
+ end
107
+ end
108
+ end
data/lib/sweat_shop.rb ADDED
@@ -0,0 +1,179 @@
1
+ require 'rubygems'
2
+ require 'digest'
3
+ require 'yaml'
4
+
5
+ $:.unshift(File.dirname(__FILE__))
6
+ require 'message_queue/base'
7
+ require 'message_queue/rabbit'
8
+ require 'message_queue/kestrel'
9
+ require 'sweat_shop/worker'
10
+
11
+ module SweatShop
12
+ extend self
13
+
14
+ def workers
15
+ @workers ||= []
16
+ end
17
+
18
+ def workers=(workers)
19
+ @workers = workers
20
+ end
21
+
22
+ def workers_in_group(groups)
23
+ groups = [groups] unless groups.is_a?(Array)
24
+ if groups.include?(:all)
25
+ workers
26
+ else
27
+ workers.select do |worker|
28
+ groups.include?(worker.queue_group)
29
+ end
30
+ end
31
+ end
32
+
33
+ def do_tasks(workers)
34
+ if queue.subscribe?
35
+ EM.run do
36
+ workers.each do |worker|
37
+ worker.subscribe
38
+ end
39
+ end
40
+ else
41
+ loop do
42
+ wait = true
43
+ workers.each do |worker|
44
+ if task = worker.dequeue
45
+ worker.do_task(task)
46
+ wait = false
47
+ end
48
+ end
49
+ if stop?
50
+ workers.each do |worker|
51
+ worker.stop
52
+ end
53
+ queue.stop
54
+ exit
55
+ end
56
+ sleep 1 if wait
57
+ end
58
+ end
59
+ end
60
+
61
+ def do_all_tasks
62
+ do_tasks(
63
+ workers_in_group(:all)
64
+ )
65
+ end
66
+
67
+ def do_default_tasks
68
+ do_tasks(
69
+ workers_in_group(:default)
70
+ )
71
+ end
72
+
73
+ def config
74
+ @config ||= begin
75
+ defaults = YAML.load_file(File.dirname(__FILE__) + '/../config/defaults.yml')
76
+ if defined?(RAILS_ROOT)
77
+ file = RAILS_ROOT + '/config/sweatshop.yml'
78
+ if File.exist?(file)
79
+ YAML.load_file(file)[RAILS_ENV || 'development']
80
+ else
81
+ defaults['enable'] = false
82
+ defaults
83
+ end
84
+ else
85
+ defaults
86
+ end
87
+ end
88
+ end
89
+
90
+ def stop
91
+ @stop = true
92
+ queue.stop if queue.subscribe?
93
+ end
94
+
95
+ def stop?
96
+ @stop
97
+ end
98
+
99
+ def queue_sizes
100
+ workers.inject([]) do |all, worker|
101
+ all << [worker, worker.queue_size]
102
+ all
103
+ end
104
+ end
105
+
106
+ def queue(type = 'default')
107
+ type = config[type] ? type : 'default'
108
+ return queues[type] if queues[type]
109
+
110
+ qconfig = config[type]
111
+ qtype = qconfig['queue'] || 'rabbit'
112
+ queue = constantize("MessageQueue::#{qtype.capitalize}")
113
+
114
+ queues[type] = queue.new(qconfig)
115
+ end
116
+
117
+ def queue=(queue, type = 'default')
118
+ queues[type] = queue
119
+ end
120
+
121
+ def queues
122
+ @queues ||= {}
123
+ end
124
+
125
+ def queue_groups
126
+ @queue_groups ||= workers.collect{|w| w.queue_group} << 'default'
127
+ end
128
+
129
+ def pp_sizes
130
+ max_width = workers.collect{|w| w.to_s.size}.max
131
+ puts '-' * (max_width + 10)
132
+ puts queue_sizes.collect{ |p| sprintf("%-#{max_width}s %2s", p.first, p.last) }.join("\n")
133
+ puts '-' * (max_width + 10)
134
+ end
135
+
136
+ def cluster_info
137
+ servers = []
138
+ queue_groups.each do |group|
139
+ qconfig = config[group]
140
+ next unless qconfig
141
+ next unless qconfig['cluster']
142
+ servers << qconfig['cluster']
143
+ end
144
+ servers.flatten!
145
+
146
+ servers.each do |server|
147
+ puts "\nQueue sizes on #{server}"
148
+ queue = MessageQueue::Rabbit.new('host' => server)
149
+ queue_groups.each do |group|
150
+ queues[group] = queue
151
+ end
152
+ pp_sizes
153
+ puts
154
+ end
155
+ @queues = {}
156
+ nil
157
+ end
158
+
159
+ def log(msg)
160
+ return if logger == :silent
161
+ logger ? logger.debug(msg) : puts(msg)
162
+ end
163
+
164
+ def logger
165
+ @logger
166
+ end
167
+
168
+ def logger=(logger)
169
+ @logger = logger
170
+ end
171
+
172
+ def constantize(str)
173
+ Object.module_eval("#{str}", __FILE__, __LINE__)
174
+ end
175
+ end
176
+
177
+ if defined?(RAILS_ROOT)
178
+ Dir.glob(RAILS_ROOT + '/app/workers/*.rb').each{|worker| require worker }
179
+ end