netinlet-sweat_shop 1.1.0
Sign up to get free protection for your applications and to get access to all the features.
- data/History.txt +6 -0
- data/LICENSE +20 -0
- data/README.markdown +105 -0
- data/Rakefile +42 -0
- data/VERSION.yml +4 -0
- data/bin/sweatd +3 -0
- data/config/defaults.yml +10 -0
- data/config/sweatshop.yml +18 -0
- data/lib/message_queue/base.rb +17 -0
- data/lib/message_queue/kestrel.rb +32 -0
- data/lib/message_queue/rabbit.rb +72 -0
- data/lib/serializers/json_serializer.rb +39 -0
- data/lib/serializers/marshal_serializer.rb +15 -0
- data/lib/serializers/serializer.rb +108 -0
- data/lib/serializers/yaml_serializer.rb +22 -0
- data/lib/sweat_shop.rb +149 -0
- data/lib/sweat_shop/daemoned.rb +405 -0
- data/lib/sweat_shop/metaid.rb +5 -0
- data/lib/sweat_shop/sweatd.rb +76 -0
- data/lib/sweat_shop/worker.rb +152 -0
- data/test/hello_worker.rb +13 -0
- data/test/test_functional_worker.rb +35 -0
- data/test/test_helper.rb +14 -0
- data/test/test_json_serializer.rb +50 -0
- data/test/test_marshal_serializer.rb +42 -0
- data/test/test_serializer.rb +95 -0
- data/test/test_sweatshop.rb +65 -0
- data/test/test_yaml_serializer.rb +44 -0
- metadata +97 -0
data/History.txt
ADDED
data/LICENSE
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
Copyright (c) 2009 Amos Elliston
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
4
|
+
a copy of this software and associated documentation files (the
|
5
|
+
"Software"), to deal in the Software without restriction, including
|
6
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
7
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
8
|
+
permit persons to whom the Software is furnished to do so, subject to
|
9
|
+
the following conditions:
|
10
|
+
|
11
|
+
The above copyright notice and this permission notice shall be
|
12
|
+
included in all copies or substantial portions of the Software.
|
13
|
+
|
14
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
15
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
16
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
17
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
18
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
19
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
20
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.markdown
ADDED
@@ -0,0 +1,105 @@
|
|
1
|
+
# SweatShop
|
2
|
+
|
3
|
+
SweatShop provides an api to background resource intensive tasks. Much of the api design was copied from Workling, with a few tweaks.
|
4
|
+
Currently, it runs rabbitmq and kestrel, but it can support any number of queues.
|
5
|
+
|
6
|
+
## Installing
|
7
|
+
|
8
|
+
gem install sweat_shop
|
9
|
+
freeze in your rails directory
|
10
|
+
cd vendor/gems/sweat_shop
|
11
|
+
rake setup
|
12
|
+
|
13
|
+
## Writing workers
|
14
|
+
|
15
|
+
Put `email_worker.rb` into app/workers and sublcass `SweatShop::Worker`:
|
16
|
+
|
17
|
+
class EmailWorker
|
18
|
+
def send_mail(to)
|
19
|
+
user = User.find_by_id(to)
|
20
|
+
Mailer.deliver_welcome(to)
|
21
|
+
end
|
22
|
+
end
|
23
|
+
|
24
|
+
Then, anywhere in your app you can execute:
|
25
|
+
|
26
|
+
EmailWorker.async_send_mail(1)
|
27
|
+
|
28
|
+
The `async` signifies that this task will be placed on a queue to be serviced by the EmailWorker possibly on another machine. You can also
|
29
|
+
call:
|
30
|
+
|
31
|
+
EmailWorker.send_mail(1)
|
32
|
+
|
33
|
+
That will do the work immediately, without placing the task on the queue. You can also define a `queue_group` at the top of the file
|
34
|
+
which will allow you to split workers out into logical groups. This is important if you have various machines serving different
|
35
|
+
queues.
|
36
|
+
|
37
|
+
## Running the queue
|
38
|
+
|
39
|
+
SweatShop has been tested with Rabbit and Kestrel, but it will also work with Starling. Please use the following resources to install the server:
|
40
|
+
|
41
|
+
Kestrel:
|
42
|
+
http://github.com/robey/kestrel/tree/master
|
43
|
+
|
44
|
+
Rabbit:
|
45
|
+
http://github.com/ezmobius/nanite/tree/master
|
46
|
+
|
47
|
+
config/sweatshop.yml specifies the machine address of the queue (default localhost:5672). You can also specify the queue type with the queue param.
|
48
|
+
|
49
|
+
## Running the workers
|
50
|
+
|
51
|
+
Assuming you ran `rake setup` in Rails, you can type:
|
52
|
+
|
53
|
+
script/sweatshop
|
54
|
+
|
55
|
+
By default, the script will run all workers defined in the app/workers dir. Every task will be processed on each queue using a round-robin algorithm. You can also add the `-d` flag which will put the worker in daemon mode. The daemon also takes other params. Add a `-h` for more details.
|
56
|
+
|
57
|
+
script/sweatshop -d
|
58
|
+
script/sweatshop -d stop
|
59
|
+
|
60
|
+
If you would like to run SweatShop as a daemon on a linux machine, use the initd.sh script provided in the sweat_shop/script dir.
|
61
|
+
|
62
|
+
# Serializers
|
63
|
+
|
64
|
+
Serializers can be swapped out on a per worker basis and set globally to a default via the config file.
|
65
|
+
|
66
|
+
The default serializer is Marshal to maintain backward compatibility with existing apps. Three serializers
|
67
|
+
are available out of the box, Marshal, JSON & YAML.
|
68
|
+
|
69
|
+
To change the default serializer, add a key [serializer] to the config file with a value of %w{marshal json yaml} or your
|
70
|
+
own custom serializer.
|
71
|
+
|
72
|
+
You may also write your own custom serializer classes by extending SweatShop::Serializer.
|
73
|
+
|
74
|
+
You can give your serializers an arbitrary class name, but the default pattern is to name it <somename>Serializer. Notice
|
75
|
+
you can specify the Marshal, JSON or YAML serializer with the name :marshal, :json, or :yaml. This default short name
|
76
|
+
is generated by underscoring the class name and removing the "Serializer" from the end of the class name, i.e.
|
77
|
+
SweatShop::Serializers::JsonSerializer becomes :json and My::Custom::EncryptedSerializer would become :encrypted. You can
|
78
|
+
override this default generated name by using the declaration in the top of your class:
|
79
|
+
serializer_name :foo
|
80
|
+
|
81
|
+
Where :foo is what you want to register the serializer as.
|
82
|
+
|
83
|
+
This is a complete example of how to write a custom serializer:
|
84
|
+
|
85
|
+
class MyCustomSerializer < SweatShop::Serializer
|
86
|
+
serializer_name :foo # overrides the generated default name of :my_custom based on the above rules
|
87
|
+
|
88
|
+
def self.serialize(data)
|
89
|
+
... # custom serialize the data
|
90
|
+
end
|
91
|
+
|
92
|
+
def self.deserialize(data)
|
93
|
+
... # custom deserialize the data
|
94
|
+
end
|
95
|
+
end
|
96
|
+
|
97
|
+
# REQUIREMENTS
|
98
|
+
|
99
|
+
i_can_daemonize
|
100
|
+
memcache (for kestrel)
|
101
|
+
carrot (for rabbit)
|
102
|
+
|
103
|
+
# LICENSE
|
104
|
+
|
105
|
+
Copyright (c) 2009 Amos Elliston, Geni.com; Published under The MIT License, see License
|
data/Rakefile
ADDED
@@ -0,0 +1,42 @@
|
|
1
|
+
require 'rake'
|
2
|
+
require 'rake/testtask'
|
3
|
+
require 'rake/rdoctask'
|
4
|
+
require 'rcov/rcovtask'
|
5
|
+
|
6
|
+
begin
|
7
|
+
require 'jeweler'
|
8
|
+
Jeweler::Tasks.new do |s|
|
9
|
+
s.name = "sweat_shop"
|
10
|
+
s.summary = %Q{SweatShop is a simple asynchronous worker queue build on top of rabbitmq/ampq}
|
11
|
+
s.email = "doug@netinlet.com"
|
12
|
+
s.homepage = "http://github.com/netinlet/sweat_shop"
|
13
|
+
s.description = "Amos Elliston's sweat_shop + Custom Serialization"
|
14
|
+
s.authors = ["Amos Elliston", "Doug Bryant"]
|
15
|
+
s.files = FileList["[A-Z]*", "{lib,test,config}/**/*"]
|
16
|
+
s.add_dependency('famoseagle-carrot', '= 0.7.0')
|
17
|
+
end
|
18
|
+
rescue LoadError
|
19
|
+
puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
|
20
|
+
end
|
21
|
+
|
22
|
+
Rake::TestTask.new
|
23
|
+
|
24
|
+
Rake::RDocTask.new do |rdoc|
|
25
|
+
rdoc.rdoc_dir = 'rdoc'
|
26
|
+
rdoc.title = 'new_project'
|
27
|
+
rdoc.options << '--line-numbers' << '--inline-source'
|
28
|
+
rdoc.rdoc_files.include('README*')
|
29
|
+
rdoc.rdoc_files.include('lib/**/*.rb')
|
30
|
+
end
|
31
|
+
|
32
|
+
Rcov::RcovTask.new do |t|
|
33
|
+
t.libs << 'test'
|
34
|
+
t.test_files = FileList['test/**/*_test.rb']
|
35
|
+
t.verbose = true
|
36
|
+
end
|
37
|
+
|
38
|
+
task :default => :test
|
39
|
+
|
40
|
+
task :setup do
|
41
|
+
require File.dirname(__FILE__) + '/install'
|
42
|
+
end
|
data/VERSION.yml
ADDED
data/bin/sweatd
ADDED
data/config/defaults.yml
ADDED
@@ -0,0 +1,18 @@
|
|
1
|
+
development:
|
2
|
+
default:
|
3
|
+
queue: rabbit
|
4
|
+
host: localhost
|
5
|
+
port: 5672
|
6
|
+
enable: true
|
7
|
+
test:
|
8
|
+
default:
|
9
|
+
queue: rabbit
|
10
|
+
host: localhost
|
11
|
+
port: 5672
|
12
|
+
enable: true
|
13
|
+
production:
|
14
|
+
default:
|
15
|
+
queue: rabbit
|
16
|
+
host: localhost
|
17
|
+
port: 5672
|
18
|
+
enable: true
|
@@ -0,0 +1,17 @@
|
|
1
|
+
module MessageQueue
|
2
|
+
class Base
|
3
|
+
attr_reader :opts
|
4
|
+
def queue_size(queue); end
|
5
|
+
def enqueue(queue, data, serializer); end
|
6
|
+
def dequeue(queue, serializer); end
|
7
|
+
def confirm(queue); end
|
8
|
+
def subscribe(queue); end
|
9
|
+
def delete(queue); end
|
10
|
+
def client; end
|
11
|
+
def stop; end
|
12
|
+
|
13
|
+
def subscribe?
|
14
|
+
false
|
15
|
+
end
|
16
|
+
end
|
17
|
+
end
|
@@ -0,0 +1,32 @@
|
|
1
|
+
module MessageQueue
|
2
|
+
class Kestrel < Base
|
3
|
+
def initialize(opts)
|
4
|
+
@servers = opts['servers']
|
5
|
+
end
|
6
|
+
|
7
|
+
def queue_size(queue)
|
8
|
+
size = 0
|
9
|
+
stats = client.stats
|
10
|
+
servers.each do |server|
|
11
|
+
size += stats[server]["queue_#{queue}_items"].to_i
|
12
|
+
end
|
13
|
+
size
|
14
|
+
end
|
15
|
+
|
16
|
+
def enqueue(queue, data, serializer)
|
17
|
+
client.set(queue, data)
|
18
|
+
end
|
19
|
+
|
20
|
+
def dequeue(queue, serializer)
|
21
|
+
client.get("#{queue}/open")
|
22
|
+
end
|
23
|
+
|
24
|
+
def confirm(queue)
|
25
|
+
client.get("#{queue}/close")
|
26
|
+
end
|
27
|
+
|
28
|
+
def client
|
29
|
+
@client ||= MemCache.new(servers)
|
30
|
+
end
|
31
|
+
end
|
32
|
+
end
|
@@ -0,0 +1,72 @@
|
|
1
|
+
require 'carrot'
|
2
|
+
module MessageQueue
|
3
|
+
class Rabbit < Base
|
4
|
+
|
5
|
+
def initialize(opts={})
|
6
|
+
@opts = opts
|
7
|
+
end
|
8
|
+
|
9
|
+
def delete(queue)
|
10
|
+
send_command do
|
11
|
+
client.queue(queue).delete
|
12
|
+
end
|
13
|
+
end
|
14
|
+
|
15
|
+
def queue_size(queue)
|
16
|
+
send_command do
|
17
|
+
client.queue(queue).message_count
|
18
|
+
end
|
19
|
+
end
|
20
|
+
|
21
|
+
def enqueue(queue, data, serializer)
|
22
|
+
send_command do
|
23
|
+
client.queue(queue, :durable => true).publish(serializer.serialize(data), :persistent => true)
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
def dequeue(queue, serializer)
|
28
|
+
send_command do
|
29
|
+
task = client.queue(queue).pop(:ack => true)
|
30
|
+
return unless task
|
31
|
+
serializer.deserialize(task)
|
32
|
+
end
|
33
|
+
end
|
34
|
+
|
35
|
+
def confirm(queue)
|
36
|
+
send_command do
|
37
|
+
client.queue(queue).ack
|
38
|
+
end
|
39
|
+
end
|
40
|
+
|
41
|
+
def send_command(&block)
|
42
|
+
retried = false
|
43
|
+
begin
|
44
|
+
block.call
|
45
|
+
rescue Carrot::AMQP::Server::ServerDown => e
|
46
|
+
if not retried
|
47
|
+
puts "Error #{e.message}. Retrying..."
|
48
|
+
@client = nil
|
49
|
+
retried = true
|
50
|
+
retry
|
51
|
+
else
|
52
|
+
raise e
|
53
|
+
end
|
54
|
+
end
|
55
|
+
end
|
56
|
+
|
57
|
+
def client
|
58
|
+
@client ||= Carrot.new(
|
59
|
+
:host => @opts['host'],
|
60
|
+
:port => @opts['port'].to_i,
|
61
|
+
:user => @opts['user'],
|
62
|
+
:pass => @opts['pass'],
|
63
|
+
:vhost => @opts['vhost'],
|
64
|
+
:insist => @opts['insist']
|
65
|
+
)
|
66
|
+
end
|
67
|
+
|
68
|
+
def stop
|
69
|
+
client.stop
|
70
|
+
end
|
71
|
+
end
|
72
|
+
end
|
@@ -0,0 +1,39 @@
|
|
1
|
+
gem 'json'
|
2
|
+
require 'json'
|
3
|
+
require 'json/add/rails'
|
4
|
+
require 'json/add/core'
|
5
|
+
|
6
|
+
module SweatShop
|
7
|
+
module Serializers
|
8
|
+
class JsonSerializer < SweatShop::Serializer
|
9
|
+
class << self
|
10
|
+
|
11
|
+
def serialize(payload)
|
12
|
+
if payload.respond_to?(:to_json)
|
13
|
+
payload.to_json
|
14
|
+
else
|
15
|
+
JSON.generate(payload)
|
16
|
+
end
|
17
|
+
end
|
18
|
+
|
19
|
+
def deserialize(payload)
|
20
|
+
symbolize_keys(JSON.parse(payload))
|
21
|
+
end
|
22
|
+
|
23
|
+
protected
|
24
|
+
# another straight out of rails land - ActiveSupport::CoreExtensions::Hash::Keys
|
25
|
+
def symbolize_keys(data)
|
26
|
+
if data.is_a?(Hash)
|
27
|
+
data.inject({}) do |options, (key, value)|
|
28
|
+
options[(key.to_sym rescue key) || key] = value
|
29
|
+
options
|
30
|
+
end
|
31
|
+
else
|
32
|
+
data
|
33
|
+
end
|
34
|
+
end
|
35
|
+
|
36
|
+
end
|
37
|
+
end
|
38
|
+
end
|
39
|
+
end
|
@@ -0,0 +1,108 @@
|
|
1
|
+
# Serializers can be swapped out on a per worker basis and set globally to a default via the config file.
|
2
|
+
#
|
3
|
+
# The default serializer is Marshal to maintain backward compatibility with existing apps. Three serializers
|
4
|
+
# are available out of the box, Marshal, JSON & YAML.
|
5
|
+
#
|
6
|
+
# To change the default serializer, add a key [serializer] to the config file with a value of %w{marshal json yaml} or your
|
7
|
+
# own custom serializer.
|
8
|
+
#
|
9
|
+
# You may also write your own custom serializer classes by extending SweatShop::Serializer.
|
10
|
+
#
|
11
|
+
# You can give your serializers an arbitrary class name, but the default pattern is to name it <somename>Serializer. Notice
|
12
|
+
# you can specify the Marshal, JSON or YAML serializer with the name :marshal, :json, or :yaml. This default short name
|
13
|
+
# is generated by underscoring the class name and removing the "Serializer" from the end of the class name, i.e.
|
14
|
+
# SweatShop::Serializers::JsonSerializer becomes :json and My::Custom::EncryptedSerializer would become :encrypted. You can
|
15
|
+
# override this default generated name by using the declaration in the top of your class:
|
16
|
+
# serializer_name :foo
|
17
|
+
#
|
18
|
+
# Where :foo is what you want to register the serializer as.
|
19
|
+
#
|
20
|
+
# This is a complete example of how to write a custom serializer:
|
21
|
+
#
|
22
|
+
# class MyCustomSerializer < SweatShop::Serializer
|
23
|
+
# serializer_name :foo # overrides the generated default name of :my_custom based on the above rules
|
24
|
+
#
|
25
|
+
# def self.serialize(data)
|
26
|
+
# ... # custom serialize the data
|
27
|
+
# end
|
28
|
+
#
|
29
|
+
# def self.deserialize(data)
|
30
|
+
# ... # custom deserialize the data
|
31
|
+
# end
|
32
|
+
# end
|
33
|
+
|
34
|
+
module SweatShop
|
35
|
+
class Serializer
|
36
|
+
|
37
|
+
def Serializer.inherited(subklass)
|
38
|
+
register(subklass.get_name, subklass)
|
39
|
+
end
|
40
|
+
|
41
|
+
def Serializer.serializers
|
42
|
+
@serializers ||= {}
|
43
|
+
end
|
44
|
+
|
45
|
+
def Serializer.default
|
46
|
+
@default_serializer ||= begin
|
47
|
+
if s = config['serializer']
|
48
|
+
unless serializers.has_key?(s.to_sym)
|
49
|
+
raise RuntimeError, "Unknown serializer [#{s}]"
|
50
|
+
end
|
51
|
+
log("Using [:#{s}] as the Default Serializer")
|
52
|
+
serializers[s.to_sym]
|
53
|
+
else
|
54
|
+
log("No Default Serializer specified. Using [:marshal]")
|
55
|
+
serializers[:marshal]
|
56
|
+
end
|
57
|
+
end
|
58
|
+
end
|
59
|
+
|
60
|
+
def Serializer.default=(serializer_name)
|
61
|
+
@default_serializer = serializers[serializer_name.to_sym]
|
62
|
+
end
|
63
|
+
|
64
|
+
def self.serializer_name(name)
|
65
|
+
@serializer_name = name
|
66
|
+
register(name, self)
|
67
|
+
end
|
68
|
+
|
69
|
+
def self.get_name
|
70
|
+
@serializer_name || generate_serializer_name(self.to_s)
|
71
|
+
end
|
72
|
+
|
73
|
+
def self.log(msg)
|
74
|
+
SweatShop.log(msg)
|
75
|
+
end
|
76
|
+
|
77
|
+
private
|
78
|
+
|
79
|
+
def Serializer.register(name, klass)
|
80
|
+
# classes get inherited before the name is set - update it by removing first
|
81
|
+
Serializer.serializers.delete(generate_serializer_name(klass.to_s))
|
82
|
+
Serializer.serializers[name] = klass
|
83
|
+
end
|
84
|
+
|
85
|
+
def self.config
|
86
|
+
SweatShop.config
|
87
|
+
end
|
88
|
+
|
89
|
+
|
90
|
+
def self.generate_serializer_name(name)
|
91
|
+
underscore(name.to_s.gsub(/Serializer$/, "")).split("/").last.to_sym
|
92
|
+
end
|
93
|
+
|
94
|
+
# Taken straight out of the rails Inflector module
|
95
|
+
def self.underscore(camel_cased_word)
|
96
|
+
camel_cased_word.to_s.gsub(/::/, '/').
|
97
|
+
gsub(/([A-Z]+)([A-Z][a-z])/,'\1_\2').
|
98
|
+
gsub(/([a-z\d])([A-Z])/,'\1_\2').
|
99
|
+
tr("-", "_").
|
100
|
+
downcase
|
101
|
+
end
|
102
|
+
end
|
103
|
+
end
|
104
|
+
|
105
|
+
|
106
|
+
require File.dirname(__FILE__) + '/marshal_serializer'
|
107
|
+
require File.dirname(__FILE__) + '/json_serializer'
|
108
|
+
require File.dirname(__FILE__) + '/yaml_serializer'
|