netinlet-sweat_shop 1.1.0

Sign up to get free protection for your applications and to get access to all the features.
@@ -0,0 +1,6 @@
1
+ === 1.0.0 / 2009-01-13
2
+
3
+ * 1 major enhancement
4
+
5
+ * Birthday!
6
+
data/LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) 2009 Amos Elliston
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,105 @@
1
+ # SweatShop
2
+
3
+ SweatShop provides an api to background resource intensive tasks. Much of the api design was copied from Workling, with a few tweaks.
4
+ Currently, it runs rabbitmq and kestrel, but it can support any number of queues.
5
+
6
+ ## Installing
7
+
8
+ gem install sweat_shop
9
+ freeze in your rails directory
10
+ cd vendor/gems/sweat_shop
11
+ rake setup
12
+
13
+ ## Writing workers
14
+
15
+ Put `email_worker.rb` into app/workers and sublcass `SweatShop::Worker`:
16
+
17
+ class EmailWorker
18
+ def send_mail(to)
19
+ user = User.find_by_id(to)
20
+ Mailer.deliver_welcome(to)
21
+ end
22
+ end
23
+
24
+ Then, anywhere in your app you can execute:
25
+
26
+ EmailWorker.async_send_mail(1)
27
+
28
+ The `async` signifies that this task will be placed on a queue to be serviced by the EmailWorker possibly on another machine. You can also
29
+ call:
30
+
31
+ EmailWorker.send_mail(1)
32
+
33
+ That will do the work immediately, without placing the task on the queue. You can also define a `queue_group` at the top of the file
34
+ which will allow you to split workers out into logical groups. This is important if you have various machines serving different
35
+ queues.
36
+
37
+ ## Running the queue
38
+
39
+ SweatShop has been tested with Rabbit and Kestrel, but it will also work with Starling. Please use the following resources to install the server:
40
+
41
+ Kestrel:
42
+ http://github.com/robey/kestrel/tree/master
43
+
44
+ Rabbit:
45
+ http://github.com/ezmobius/nanite/tree/master
46
+
47
+ config/sweatshop.yml specifies the machine address of the queue (default localhost:5672). You can also specify the queue type with the queue param.
48
+
49
+ ## Running the workers
50
+
51
+ Assuming you ran `rake setup` in Rails, you can type:
52
+
53
+ script/sweatshop
54
+
55
+ By default, the script will run all workers defined in the app/workers dir. Every task will be processed on each queue using a round-robin algorithm. You can also add the `-d` flag which will put the worker in daemon mode. The daemon also takes other params. Add a `-h` for more details.
56
+
57
+ script/sweatshop -d
58
+ script/sweatshop -d stop
59
+
60
+ If you would like to run SweatShop as a daemon on a linux machine, use the initd.sh script provided in the sweat_shop/script dir.
61
+
62
+ # Serializers
63
+
64
+ Serializers can be swapped out on a per worker basis and set globally to a default via the config file.
65
+
66
+ The default serializer is Marshal to maintain backward compatibility with existing apps. Three serializers
67
+ are available out of the box, Marshal, JSON & YAML.
68
+
69
+ To change the default serializer, add a key [serializer] to the config file with a value of %w{marshal json yaml} or your
70
+ own custom serializer.
71
+
72
+ You may also write your own custom serializer classes by extending SweatShop::Serializer.
73
+
74
+ You can give your serializers an arbitrary class name, but the default pattern is to name it <somename>Serializer. Notice
75
+ you can specify the Marshal, JSON or YAML serializer with the name :marshal, :json, or :yaml. This default short name
76
+ is generated by underscoring the class name and removing the "Serializer" from the end of the class name, i.e.
77
+ SweatShop::Serializers::JsonSerializer becomes :json and My::Custom::EncryptedSerializer would become :encrypted. You can
78
+ override this default generated name by using the declaration in the top of your class:
79
+ serializer_name :foo
80
+
81
+ Where :foo is what you want to register the serializer as.
82
+
83
+ This is a complete example of how to write a custom serializer:
84
+
85
+ class MyCustomSerializer < SweatShop::Serializer
86
+ serializer_name :foo # overrides the generated default name of :my_custom based on the above rules
87
+
88
+ def self.serialize(data)
89
+ ... # custom serialize the data
90
+ end
91
+
92
+ def self.deserialize(data)
93
+ ... # custom deserialize the data
94
+ end
95
+ end
96
+
97
+ # REQUIREMENTS
98
+
99
+ i_can_daemonize
100
+ memcache (for kestrel)
101
+ carrot (for rabbit)
102
+
103
+ # LICENSE
104
+
105
+ Copyright (c) 2009 Amos Elliston, Geni.com; Published under The MIT License, see License
@@ -0,0 +1,42 @@
1
+ require 'rake'
2
+ require 'rake/testtask'
3
+ require 'rake/rdoctask'
4
+ require 'rcov/rcovtask'
5
+
6
+ begin
7
+ require 'jeweler'
8
+ Jeweler::Tasks.new do |s|
9
+ s.name = "sweat_shop"
10
+ s.summary = %Q{SweatShop is a simple asynchronous worker queue build on top of rabbitmq/ampq}
11
+ s.email = "doug@netinlet.com"
12
+ s.homepage = "http://github.com/netinlet/sweat_shop"
13
+ s.description = "Amos Elliston's sweat_shop + Custom Serialization"
14
+ s.authors = ["Amos Elliston", "Doug Bryant"]
15
+ s.files = FileList["[A-Z]*", "{lib,test,config}/**/*"]
16
+ s.add_dependency('famoseagle-carrot', '= 0.7.0')
17
+ end
18
+ rescue LoadError
19
+ puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
20
+ end
21
+
22
+ Rake::TestTask.new
23
+
24
+ Rake::RDocTask.new do |rdoc|
25
+ rdoc.rdoc_dir = 'rdoc'
26
+ rdoc.title = 'new_project'
27
+ rdoc.options << '--line-numbers' << '--inline-source'
28
+ rdoc.rdoc_files.include('README*')
29
+ rdoc.rdoc_files.include('lib/**/*.rb')
30
+ end
31
+
32
+ Rcov::RcovTask.new do |t|
33
+ t.libs << 'test'
34
+ t.test_files = FileList['test/**/*_test.rb']
35
+ t.verbose = true
36
+ end
37
+
38
+ task :default => :test
39
+
40
+ task :setup do
41
+ require File.dirname(__FILE__) + '/install'
42
+ end
@@ -0,0 +1,4 @@
1
+ ---
2
+ :major: 1
3
+ :minor: 1
4
+ :patch: 0
@@ -0,0 +1,3 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require File.dirname(__FILE__) + '/../lib/sweat_shop/sweatd'
@@ -0,0 +1,10 @@
1
+ # default options
2
+ default:
3
+ queue: rabbit
4
+ host: localhost
5
+ port: 5672
6
+ user: 'guest'
7
+ pass: 'guest'
8
+ vhost: '/'
9
+ serializer: 'marshal'
10
+ enable: true
@@ -0,0 +1,18 @@
1
+ development:
2
+ default:
3
+ queue: rabbit
4
+ host: localhost
5
+ port: 5672
6
+ enable: true
7
+ test:
8
+ default:
9
+ queue: rabbit
10
+ host: localhost
11
+ port: 5672
12
+ enable: true
13
+ production:
14
+ default:
15
+ queue: rabbit
16
+ host: localhost
17
+ port: 5672
18
+ enable: true
@@ -0,0 +1,17 @@
1
+ module MessageQueue
2
+ class Base
3
+ attr_reader :opts
4
+ def queue_size(queue); end
5
+ def enqueue(queue, data, serializer); end
6
+ def dequeue(queue, serializer); end
7
+ def confirm(queue); end
8
+ def subscribe(queue); end
9
+ def delete(queue); end
10
+ def client; end
11
+ def stop; end
12
+
13
+ def subscribe?
14
+ false
15
+ end
16
+ end
17
+ end
@@ -0,0 +1,32 @@
1
+ module MessageQueue
2
+ class Kestrel < Base
3
+ def initialize(opts)
4
+ @servers = opts['servers']
5
+ end
6
+
7
+ def queue_size(queue)
8
+ size = 0
9
+ stats = client.stats
10
+ servers.each do |server|
11
+ size += stats[server]["queue_#{queue}_items"].to_i
12
+ end
13
+ size
14
+ end
15
+
16
+ def enqueue(queue, data, serializer)
17
+ client.set(queue, data)
18
+ end
19
+
20
+ def dequeue(queue, serializer)
21
+ client.get("#{queue}/open")
22
+ end
23
+
24
+ def confirm(queue)
25
+ client.get("#{queue}/close")
26
+ end
27
+
28
+ def client
29
+ @client ||= MemCache.new(servers)
30
+ end
31
+ end
32
+ end
@@ -0,0 +1,72 @@
1
+ require 'carrot'
2
+ module MessageQueue
3
+ class Rabbit < Base
4
+
5
+ def initialize(opts={})
6
+ @opts = opts
7
+ end
8
+
9
+ def delete(queue)
10
+ send_command do
11
+ client.queue(queue).delete
12
+ end
13
+ end
14
+
15
+ def queue_size(queue)
16
+ send_command do
17
+ client.queue(queue).message_count
18
+ end
19
+ end
20
+
21
+ def enqueue(queue, data, serializer)
22
+ send_command do
23
+ client.queue(queue, :durable => true).publish(serializer.serialize(data), :persistent => true)
24
+ end
25
+ end
26
+
27
+ def dequeue(queue, serializer)
28
+ send_command do
29
+ task = client.queue(queue).pop(:ack => true)
30
+ return unless task
31
+ serializer.deserialize(task)
32
+ end
33
+ end
34
+
35
+ def confirm(queue)
36
+ send_command do
37
+ client.queue(queue).ack
38
+ end
39
+ end
40
+
41
+ def send_command(&block)
42
+ retried = false
43
+ begin
44
+ block.call
45
+ rescue Carrot::AMQP::Server::ServerDown => e
46
+ if not retried
47
+ puts "Error #{e.message}. Retrying..."
48
+ @client = nil
49
+ retried = true
50
+ retry
51
+ else
52
+ raise e
53
+ end
54
+ end
55
+ end
56
+
57
+ def client
58
+ @client ||= Carrot.new(
59
+ :host => @opts['host'],
60
+ :port => @opts['port'].to_i,
61
+ :user => @opts['user'],
62
+ :pass => @opts['pass'],
63
+ :vhost => @opts['vhost'],
64
+ :insist => @opts['insist']
65
+ )
66
+ end
67
+
68
+ def stop
69
+ client.stop
70
+ end
71
+ end
72
+ end
@@ -0,0 +1,39 @@
1
+ gem 'json'
2
+ require 'json'
3
+ require 'json/add/rails'
4
+ require 'json/add/core'
5
+
6
+ module SweatShop
7
+ module Serializers
8
+ class JsonSerializer < SweatShop::Serializer
9
+ class << self
10
+
11
+ def serialize(payload)
12
+ if payload.respond_to?(:to_json)
13
+ payload.to_json
14
+ else
15
+ JSON.generate(payload)
16
+ end
17
+ end
18
+
19
+ def deserialize(payload)
20
+ symbolize_keys(JSON.parse(payload))
21
+ end
22
+
23
+ protected
24
+ # another straight out of rails land - ActiveSupport::CoreExtensions::Hash::Keys
25
+ def symbolize_keys(data)
26
+ if data.is_a?(Hash)
27
+ data.inject({}) do |options, (key, value)|
28
+ options[(key.to_sym rescue key) || key] = value
29
+ options
30
+ end
31
+ else
32
+ data
33
+ end
34
+ end
35
+
36
+ end
37
+ end
38
+ end
39
+ end
@@ -0,0 +1,15 @@
1
+ module SweatShop
2
+ module Serializers
3
+ class MarshalSerializer < SweatShop::Serializer
4
+ class << self
5
+ def serialize(payload)
6
+ Marshal.dump(payload)
7
+ end
8
+
9
+ def deserialize(payload)
10
+ Marshal.load(payload)
11
+ end
12
+ end
13
+ end
14
+ end
15
+ end
@@ -0,0 +1,108 @@
1
+ # Serializers can be swapped out on a per worker basis and set globally to a default via the config file.
2
+ #
3
+ # The default serializer is Marshal to maintain backward compatibility with existing apps. Three serializers
4
+ # are available out of the box, Marshal, JSON & YAML.
5
+ #
6
+ # To change the default serializer, add a key [serializer] to the config file with a value of %w{marshal json yaml} or your
7
+ # own custom serializer.
8
+ #
9
+ # You may also write your own custom serializer classes by extending SweatShop::Serializer.
10
+ #
11
+ # You can give your serializers an arbitrary class name, but the default pattern is to name it <somename>Serializer. Notice
12
+ # you can specify the Marshal, JSON or YAML serializer with the name :marshal, :json, or :yaml. This default short name
13
+ # is generated by underscoring the class name and removing the "Serializer" from the end of the class name, i.e.
14
+ # SweatShop::Serializers::JsonSerializer becomes :json and My::Custom::EncryptedSerializer would become :encrypted. You can
15
+ # override this default generated name by using the declaration in the top of your class:
16
+ # serializer_name :foo
17
+ #
18
+ # Where :foo is what you want to register the serializer as.
19
+ #
20
+ # This is a complete example of how to write a custom serializer:
21
+ #
22
+ # class MyCustomSerializer < SweatShop::Serializer
23
+ # serializer_name :foo # overrides the generated default name of :my_custom based on the above rules
24
+ #
25
+ # def self.serialize(data)
26
+ # ... # custom serialize the data
27
+ # end
28
+ #
29
+ # def self.deserialize(data)
30
+ # ... # custom deserialize the data
31
+ # end
32
+ # end
33
+
34
+ module SweatShop
35
+ class Serializer
36
+
37
+ def Serializer.inherited(subklass)
38
+ register(subklass.get_name, subklass)
39
+ end
40
+
41
+ def Serializer.serializers
42
+ @serializers ||= {}
43
+ end
44
+
45
+ def Serializer.default
46
+ @default_serializer ||= begin
47
+ if s = config['serializer']
48
+ unless serializers.has_key?(s.to_sym)
49
+ raise RuntimeError, "Unknown serializer [#{s}]"
50
+ end
51
+ log("Using [:#{s}] as the Default Serializer")
52
+ serializers[s.to_sym]
53
+ else
54
+ log("No Default Serializer specified. Using [:marshal]")
55
+ serializers[:marshal]
56
+ end
57
+ end
58
+ end
59
+
60
+ def Serializer.default=(serializer_name)
61
+ @default_serializer = serializers[serializer_name.to_sym]
62
+ end
63
+
64
+ def self.serializer_name(name)
65
+ @serializer_name = name
66
+ register(name, self)
67
+ end
68
+
69
+ def self.get_name
70
+ @serializer_name || generate_serializer_name(self.to_s)
71
+ end
72
+
73
+ def self.log(msg)
74
+ SweatShop.log(msg)
75
+ end
76
+
77
+ private
78
+
79
+ def Serializer.register(name, klass)
80
+ # classes get inherited before the name is set - update it by removing first
81
+ Serializer.serializers.delete(generate_serializer_name(klass.to_s))
82
+ Serializer.serializers[name] = klass
83
+ end
84
+
85
+ def self.config
86
+ SweatShop.config
87
+ end
88
+
89
+
90
+ def self.generate_serializer_name(name)
91
+ underscore(name.to_s.gsub(/Serializer$/, "")).split("/").last.to_sym
92
+ end
93
+
94
+ # Taken straight out of the rails Inflector module
95
+ def self.underscore(camel_cased_word)
96
+ camel_cased_word.to_s.gsub(/::/, '/').
97
+ gsub(/([A-Z]+)([A-Z][a-z])/,'\1_\2').
98
+ gsub(/([a-z\d])([A-Z])/,'\1_\2').
99
+ tr("-", "_").
100
+ downcase
101
+ end
102
+ end
103
+ end
104
+
105
+
106
+ require File.dirname(__FILE__) + '/marshal_serializer'
107
+ require File.dirname(__FILE__) + '/json_serializer'
108
+ require File.dirname(__FILE__) + '/yaml_serializer'