donkey 0.1.0

Sign up to get free protection for your applications and to get access to all the features.
data/LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) 2009 Howard Yeh
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,162 @@
1
+
2
+ Asynchronous Service Stages (ASS) is a way to
3
+ organize distributed services by decoupling the
4
+ what and how of computation from the when and
5
+ where. Built on top of RabbitMQ (an implementation
6
+ of AMQP), ASS helps you to build robust and
7
+ scalable distributed applications.
8
+
9
+ End of project pimping, let's get started.
10
+
11
+ h1. Install
12
+
13
+ You need
14
+
15
+ (1) Erlang
16
+ (2) RabbitMQ
17
+ (3) AMQP gem
18
+
19
+ Thread.new { EM.run }
20
+ AMQP.start
21
+
22
+ ^C to exit
23
+
24
+
25
+ h1. The Basics
26
+
27
+ A service component is a ruby script that
28
+ communicates with RabbitMQ. You need to define the
29
+ AMQP server your ASS depends on. Something like this,
30
+
31
+
32
+ require 'rubygems'
33
+ require 'ass'
34
+ AMQP.start(:host => 'localhost',
35
+ #:vhost => "/ass-test",
36
+ :logging => false) do
37
+ # ASS definition
38
+ end
39
+
40
+
41
+ To start a server
42
+
43
+ server = ASS.new("echo")
44
+ # => #<ASS::Server echo>
45
+
46
+ But it doesn't do anything yet. You define the
47
+ behaviour of the server by setting its
48
+ callback. The callback can be a class, so that for
49
+ each client request an object is created from the
50
+ class to process the request. Like so,
51
+
52
+
53
+ server.react(SomeReactorClass)
54
+
55
+
56
+ However, often you just want something simple. The
57
+ react method can take a block and construct an
58
+ anonymous callback class from which the server
59
+ creates an callback object for each request. Here
60
+ we ask the server to react to @foo@ or @bar@.
61
+
62
+
63
+ server.react {
64
+ def foo(input)
65
+ [:server,:foo,input]
66
+ end
67
+
68
+ def oof(input)
69
+ [:server,:oof,input]
70
+ end
71
+ }
72
+
73
+
74
+ The react method accepts for the callback either
75
+ a Class, a Module, a block, or any object. When an
76
+ object is used, it's considered a singleton, which
77
+ is used to process all the requests.
78
+
79
+ Now that we have a server, we need to get a client
80
+ so to call the server. Because the call is
81
+ asynchronous (the client doesn't wait for the
82
+ result), to process the result when it gets back,
83
+ we need to define callback for the client (just as
84
+ we did for the server). For each call to the
85
+ remote server, the result is processed at the
86
+ client side by a method of the same name,
87
+
88
+
89
+ client = server.client.react {
90
+ def foo(output)
91
+ p [:client,:foo,output]
92
+ end
93
+ def oof(output)
94
+ p [:client,:oof,output]
95
+ end
96
+ }
97
+
98
+ c.call(:foo,42)
99
+ c.call(:oof,24)
100
+
101
+ # [:client,:foo,[:server,:foo,42]]
102
+ # [:client,:foo,[:server,:foo,24]]
103
+
104
+
105
+ > ruby server.rb
106
+ > ruby client.rb
107
+
108
+ > ruby server.rb
109
+ ^C
110
+
111
+ While the server is down, the requests the client
112
+ is making is queued by the underlying message
113
+ middleware (RabbitMQ), so in some future time when
114
+ we restart the server, we wouldn't lose any
115
+ request. Let's restart the server.
116
+
117
+ > ruby server.rb
118
+
119
+ See that the server caught up with all the pending
120
+ requests. To increase service capacity, we can
121
+ just increase the number of server instances.
122
+
123
+ > ruby server.rb
124
+
125
+ Now the load is distributed between these two
126
+ instances. We can also start more clients to
127
+ handle more load.
128
+
129
+ > ruby client.rb
130
+
131
+ You can see requests coming in from two clients.
132
+
133
+
134
+ h1. Service Configuration
135
+
136
+ -how the name of a service map to AMQP entities.
137
+ -various options for different functional characteristics.
138
+
139
+ -using routing_key
140
+ -using reply_to
141
+
142
+
143
+ rabbitmqctl list_exchanges
144
+ rabbitmqctl list_queues
145
+
146
+
147
+ h1. RPC service
148
+
149
+ The RPC client provides a synchronous API to the
150
+ asynchronous services. This is so the users of an
151
+ ASS application don't have to bother with the
152
+ difficulties of asynchronous programming style
153
+ with callbacks.
154
+
155
+ The RPC client is intended to be used as the
156
+ gateway into some reliable internal
157
+ services. While the internal services coordinated
158
+ with ASS needs to be robust to component failures,
159
+ there's no such requirements for gateways. It is
160
+ ok for a gateway to fail, and fail in delivering a
161
+ response, as long as the internal services carry
162
+ out the task without a hitch.
@@ -0,0 +1,57 @@
1
+ require 'rubygems'
2
+ require 'rake'
3
+
4
+ begin
5
+ require 'jeweler'
6
+ Jeweler::Tasks.new do |gem|
7
+ gem.name = "donkey"
8
+ gem.summary = "Asynchronous Service Stages for Distributed Services"
9
+ gem.email = "hayeah@gmail.com"
10
+ gem.homepage = "http://github.com/hayeah/donkey"
11
+ gem.authors = ["Howard Yeh"]
12
+ gem.add_dependency "amqp"
13
+ gem.files = FileList["[A-Z]*", "{lib,test}/**/*"]
14
+ # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
15
+ end
16
+ rescue LoadError
17
+ puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
18
+ end
19
+
20
+ require 'rake/testtask'
21
+ Rake::TestTask.new(:test) do |test|
22
+ test.libs << 'lib' << 'test'
23
+ test.pattern = 'test/**/*_test.rb'
24
+ test.verbose = true
25
+ end
26
+
27
+ begin
28
+ require 'rcov/rcovtask'
29
+ Rcov::RcovTask.new do |test|
30
+ test.libs << 'test'
31
+ test.pattern = 'test/**/*_test.rb'
32
+ test.verbose = true
33
+ end
34
+ rescue LoadError
35
+ task :rcov do
36
+ abort "RCov is not available. In order to run rcov, you must: sudo gem install spicycode-rcov"
37
+ end
38
+ end
39
+
40
+
41
+ task :default => :test
42
+
43
+ require 'rake/rdoctask'
44
+ Rake::RDocTask.new do |rdoc|
45
+ if File.exist?('VERSION.yml')
46
+ config = YAML.load(File.read('VERSION.yml'))
47
+ version = "#{config[:major]}.#{config[:minor]}.#{config[:patch]}"
48
+ else
49
+ version = ""
50
+ end
51
+
52
+ rdoc.rdoc_dir = 'rdoc'
53
+ rdoc.title = "ass #{version}"
54
+ rdoc.rdoc_files.include('README*')
55
+ rdoc.rdoc_files.include('lib/**/*.rb')
56
+ end
57
+
@@ -0,0 +1,4 @@
1
+ ---
2
+ :major: 0
3
+ :minor: 1
4
+ :patch: 0
@@ -0,0 +1,125 @@
1
+ $:.unshift File.expand_path(File.dirname(File.expand_path(__FILE__)))
2
+ require 'mq'
3
+
4
+ module ASS; end
5
+ require 'ass/amqp' # monkey patch stolen from nanite.
6
+ require 'ass/serializers'
7
+
8
+ require 'ass/server' # monkey patch stolen from nanite.
9
+ require 'ass/callback_factory'
10
+ require 'ass/actor'
11
+ require 'ass/rpc'
12
+ require 'ass/client'
13
+
14
+ require 'ass/topic'
15
+
16
+ module ASS
17
+
18
+ class << self
19
+
20
+ #MQ = nil
21
+ def start(settings={})
22
+ raise "should have one ASS per eventmachine" if EM.reactor_running? == true # allow ASS to restart if EM is not running.
23
+ EM.run {
24
+ @serializer = settings.delete(:format) || ::Marshal
25
+ raise "Object Serializer must respond to :load and :dump" unless @serializer.respond_to?(:load) && @serializer.respond_to?(:dump)
26
+ @mq = ::MQ.new(AMQP.start(settings))
27
+ # ASS and its worker threads (EM.threadpool) should share the same MQ instance.
28
+ yield if block_given?
29
+ }
30
+ end
31
+
32
+ def stop
33
+ AMQP.stop{ EM.stop }
34
+ true
35
+ end
36
+
37
+ def mq
38
+ @mq
39
+ end
40
+
41
+ def serializer
42
+ @serializer
43
+ end
44
+
45
+ def server(name,opts={},&block)
46
+ s = ASS::Server.new(name,opts)
47
+ if block
48
+ s.react(&block)
49
+ end
50
+ s
51
+ end
52
+
53
+ def actor(name,opts={},&block)
54
+ s = ASS::Actor.new(name,opts)
55
+ if block
56
+ s.react(&block)
57
+ end
58
+ s
59
+ end
60
+
61
+ def rpc(opts={})
62
+ ASS::RPC.new(opts)
63
+ end
64
+
65
+ # the opts is used to initiate an RPC
66
+ def client(opts={})
67
+ ASS::Client.new(opts)
68
+ end
69
+
70
+ # maybe move cast and call into ASS::Server's class methods
71
+ def cast(name,method,data,opts,meta)
72
+ call(name,method,data,opts.merge(:reply_to => nil),meta)
73
+ end
74
+
75
+ def call(name,method,data,opts,meta)
76
+ # make sure the payload hash use string
77
+ # keys. Serialization format might not
78
+ # preserve type.
79
+ payload = {
80
+ #:type => type,
81
+ "method" => method,
82
+ "data" => data,
83
+ "meta" => meta,
84
+ }
85
+ payload.merge("version" => opts[:version]) if opts.has_key?(:version)
86
+ payload.merge("meta" => opts[:meta]) if opts.has_key?(:meta)
87
+ dummy_exchange(name).publish(ASS.serializer.dump(payload),opts)
88
+ true
89
+ end
90
+
91
+ # this would create a dummy MQ exchange object
92
+ # for the sole purpose of publishing the
93
+ # message. Will not clobber existing server
94
+ # already started in the process.
95
+ def dummy_exchange(name)
96
+ @mq.direct(name,:no_declare => true)
97
+ end
98
+ end
99
+
100
+
101
+ # def self.topic(name,opts={})
102
+ # ASS::Topic.new(name,opts)
103
+ # end
104
+
105
+
106
+
107
+
108
+ # def self.peep(server_name,callback=nil,&block)
109
+ # callback = block if callback.nil?
110
+ # callback = Module.new {
111
+ # def server(*args)
112
+ # p [:server,args]
113
+ # end
114
+
115
+ # def client(*args)
116
+ # p [:client,args]
117
+ # end
118
+ # }
119
+ # ASS::Peeper.new(server_name,callback)
120
+ # end
121
+
122
+ # assumes server initializes it with an exclusive and auto_delete queue.
123
+
124
+
125
+ end
@@ -0,0 +1,50 @@
1
+ class ASS::Actor
2
+ # this class is a thin layer over ASS::Server
3
+ def initialize(name,opts={},&block)
4
+ @server = ASS.server(name,opts)
5
+ if block
6
+ react(&block)
7
+ end
8
+ end
9
+
10
+ def queue(opts={})
11
+ @server.queue(opts)
12
+ self
13
+ end
14
+
15
+ def react(callback=nil,opts=nil,&block)
16
+ if block
17
+ opts = callback
18
+ callback = block
19
+ end
20
+ opts = {} if opts.nil?
21
+ callback_factory = ASS::CallbackFactory.new(callback)
22
+ server = @server # for closure capturing, needed for callback_factory
23
+ @server.react(opts) {
24
+ define_method(:on_call) do |_|
25
+ raise "can't call an actor with method set to nil" if payload["method"].nil?
26
+ callback_object = callback_factory.callback_for(server,header,payload)
27
+ callback_object.send(payload["method"],payload["data"])
28
+ end
29
+
30
+ define_method(:on_error) do |e,_|
31
+ callback_object = callback_factory.callback_for(server,header,payload)
32
+ if callback_object.respond_to?(:on_error)
33
+ callback_object.on_error(e,payload["data"])
34
+ else
35
+ raise e
36
+ end
37
+ end
38
+ }
39
+ self
40
+ end
41
+
42
+ def call(*args)
43
+ @server.call(*args)
44
+ end
45
+
46
+ def cast(*args)
47
+ @server.cast(*args)
48
+ end
49
+
50
+ end
@@ -0,0 +1,19 @@
1
+
2
+ # monkey patch to the amqp gem that adds :no_declare => true option for new
3
+ # Exchange objects. This allows us to send messeages to exchanges that are
4
+ # declared by the mappers and that we have no configuration priviledges on.
5
+ # temporary until we get this into amqp proper
6
+ MQ::Exchange.class_eval do
7
+ def initialize mq, type, name, opts = {}
8
+ @mq = mq
9
+ @type, @name, @opts = type, name, opts
10
+ @mq.exchanges[@name = name] ||= self
11
+ @key = opts[:key]
12
+
13
+ @mq.callback{
14
+ @mq.send AMQP::Protocol::Exchange::Declare.new({ :exchange => name,
15
+ :type => type,
16
+ :nowait => true }.merge(opts))
17
+ } unless name == "amq.#{type}" or name == '' or opts[:no_declare]
18
+ end
19
+ end
@@ -0,0 +1,95 @@
1
+ class ASS::CallbackFactory
2
+
3
+ module ServiceMethods
4
+ def resend
5
+ throw(:__ass_resend)
6
+ end
7
+
8
+ def discard
9
+ throw(:__ass_discard)
10
+ end
11
+
12
+ def header
13
+ @__header__
14
+ end
15
+
16
+ def payload
17
+ @__payload__
18
+ end
19
+
20
+ def method
21
+ @__method__
22
+ end
23
+
24
+ def data
25
+ @__data__
26
+ end
27
+
28
+ def meta
29
+ @__meta__
30
+ end
31
+
32
+ def version
33
+ @__version__
34
+ end
35
+
36
+ def call(name,method,data=nil,opts={},meta=nil)
37
+ @__service__.call(name,method,data,opts,meta)
38
+ end
39
+
40
+ def cast(name,method,data=nil,opts={},meta=nil)
41
+ @__service__.cast(name,method,data,opts,meta)
42
+ end
43
+ end
44
+
45
+ def initialize(callback)
46
+ @factory = build_factory(callback)
47
+ end
48
+
49
+ def callback_for(server,header,payload)
50
+ # method,data
51
+ if @factory.is_a? Class
52
+ if @factory.respond_to? :version
53
+ klass = @factory.get_version(payload["version"])
54
+ else
55
+ klass = @factory
56
+ end
57
+ obj = klass.new
58
+ else
59
+ obj = @factory
60
+ end
61
+ obj.instance_variable_set("@__service__",server)
62
+ obj.instance_variable_set("@__header__",header)
63
+ obj.instance_variable_set("@__payload__",payload)
64
+ obj.instance_variable_set("@__method__",payload["method"])
65
+ obj.instance_variable_set("@__data__",payload["data"])
66
+ obj.instance_variable_set("@__meta__",payload["meta"])
67
+ obj.instance_variable_set("@__version__",payload["version"])
68
+ obj
69
+ end
70
+
71
+ private
72
+
73
+ def build_factory(callback)
74
+ c = case callback
75
+ when Proc
76
+ Class.new &callback
77
+ when Class
78
+ callback
79
+ when Module
80
+ Class.new { include callback }
81
+ else
82
+ raise "can build factory from one of Proc, Class, Module"
83
+ end
84
+ case c
85
+ when Class
86
+ c.instance_eval { include ServiceMethods }
87
+ else
88
+ c.extend ServiceMethods
89
+ end
90
+ c
91
+ end
92
+
93
+
94
+
95
+ end