adamhenry-vincent 0.0.2

data/README

@@ -0,0 +1,153 @@
+
+======= Vincent =======
+
+This a PROOF for a micro-framework for handling AMQP traffic between multiple
+daemons.  It was written with Fibers on Ruby 1.9 to allow for synchronous action
+inside event machine and MAY work in 1.8 if patched with fibers but is untested.
+The is PRE-ALPHA and it needs a full design and implementation phase as well as
+full tests and examples.  If anyone wants to contribute I am totally open.
+
+=== Background:
+
+Vincent gets you many of the AMQP benefits out of the box that most would miss
+if they do not know the protocol very well.  For instance it hoes a clean
+shutdown on a SIGTERM and transparently handles message acknowledgements for
+you so that you never lose an AMQP message.  Like-wise it includes a TTL option
+on messages that lets you kill off messages that might have stuck around beyond
+their usefulness. 
+
+=== Model:
+
+Http App Server (thin/mongrel) are not the place to be processing long running
+work (100ms max).  These app servers should be doing only simple database
+operations and displaying cached data.  When work needs to be done an
+asynchronous message should be sent to worker processes.  Vincent is broken 
+into several libraries that allow you to enforce that your app servers are 
+only sending data, while the workers can both send an receive.  
+
+Vincent needs to be run inside an Event Machine reactor.  Thin uses EventMachine.
+If you are using an app server that does not use EM, you will need to start the 
+reactor manually. 
+
+=== Client Example:
+
+I use the 'cast' and the 'call' idioms from erlang to specify if you are
+sending a message (cast) or sending a message and awaiting a reply (call).  It
+is important to not confuse these two.
+
+require 'vincent'
+
+Vincent::Client.cast("file.encrypt", { :key_id => 1234, :file => "/gfs1/file.dat", :enc => 'blowfish' })
+
+## Vincent.file_encrypt( { ... } ) does the same thing with method_missing?
+
+Cast takes a hash and will encode the that via JSON before publishing the AMQP
+message and will decode the hash on the other side.
+
+=== In line Server Example:
+
+An in line server can be setup as such:
+
+require 'vincent/server'
+
+Vincent::Server.start do |vincent|
+  vincent.cast("server.up", { :host => ENV['HOSTNAME'], :pid => Process.pid } )
+
+  vincent.listen4("file.encrypt", :queue => 'file.encrypt.queue') do |b|
+    file = get_file(b['file'])
+    key = get_key(b['enc'],b['key_id'])
+    file.lock(key)
+    b.cast("file.encrypted", { :file => file.name })
+  end
+end
+
+The code above sends a "server.up" message when it comes online and encrypts
+files when told to.  All such processes listen on the same named queue so they
+will do the work round-robin.  After finishing the file it sends a
+file.encrypted message.
+
+=== Call:
+
+Now imagine the files are stored on a file server and an AMQP message needs to
+be sent to fetch it.  This is no easy task since AMQP is asynchronous and in
+eventmachine.  Fibers come to the rescue.
+
+Vincent::Server.start do |vincent|
+  vincent.listen4("file.encrypt", :queue => 'file.encrypt.queue') do |b|
+    file = b.call("file.get", { :file => b['file'] })
+    key = get_key(b['enc'],b['key_id'])
+    file.lock(key)
+    b.cast("file.encrypted", { :file => file.name })
+  end
+end
+
+Here is the file server code - Vincent reads the file and sends its bytes
+back over the wire
+
+Vincent::Server.start do |vincent|
+  vincent.listen4("file.get") do |b|
+    File.read(b['file'])
+  end
+end
+
+Note: if the call block were to throw an exception that exception would 
+propagate up to the callers block.
+
+This process is good, but it breaks down as your app size grows.  Code is hard
+to reuse and spec, and the size of the server block will grow out of control. 
+That's where Vincent::Base comes in.
+
+=== The Vincent Object Model:
+
+require 'vincent/base'
+
+class FilePut < Vincent::Base
+  bind_to "file.put", :route_to => :one, :active => :disk_not_full
+
+  ## only one worker will receive the message
+
+  ## this class accepts requests to store files locally if and 
+  ## only if the disk is not full
+
+  def disk_not_full
+    not disk_full?
+  end
+
+  def handle_cast
+    File.write(params['file_name'],"w") do |f|
+      f.write(params['file_bytes'])
+    end
+  end
+end
+
+class FileGet < Vincent::Base
+bind_to "file.get", :route_to => :one 
+
+## this decrypts the file but rejects requests for a file that does 
+## not exist, since it probably is on a different server
+
+def handle_call
+  reject unless File.exists?(params['file'])
+    File.read(params['file'])
+  end
+end
+
+
+=== What's Next?
+
+There is a lot of work to be done:
+
+  * finalize the interface
+  * get it working on 1.8.7+fibers
+  * get full test coverage
+  * test the fiber code more - make sure it works in all cases
+  * make sure it's wire compatible with current AMQP network
+  * write examples
+  * write docs
+  * do a proper gem release
+
+=== Why call it Vincent?
+
+He was that scrappy little robot from the movie The Black Hole.  =)
+
+

data/lib/vincent/base.rb

@@ -0,0 +1,198 @@
+
+require 'vincent'
+require 'fiber'
+
+### TODO
+
+### symbolize keys
+### confirm works with droids 
+### headers
+### cache
+### autodelete queues?
+
+module Vincent
+
+  class RejectMessage < RuntimeError ; end
+  class MissingCallHandler < RuntimeError ; end
+  class MissingCastHandler < RuntimeError ; end
+
+  module Core
+    def call(key, args = {})
+      args['reply_to'] = "reply_to.#{rand 99_999_999}"
+      reply = receive(args['reply_to']) do
+        cast(key, args)
+      end
+      raise unpack(reply["exception"]) if reply["exception"]
+      return unpack(reply["results"]) if reply["results"]
+      return reply
+    end
+
+    def receive(q)
+      f = Fiber.current
+      subscribe(q) do |result|
+        ## maybe we can destroy the queue here - unsubscribe - autodelete
+        f.resume(result)
+      end
+      yield
+      Fiber.yield
+    end
+
+    def listen4(key, options = {}, &block)
+      q = options[:queue]
+      q ||= "q.#{ENV['HOSTNAME']}" 
+
+      bind(q, key)
+      subscribe(q) do |args|
+        block.call(Vincent::Base.new(args, args))
+      end
+    end
+
+    def bind(q, key)
+      MQ.queue(q).bind(exchange, :key => key)
+    end
+
+    def unsubscribe(q)
+      MQ.queue(q).unsubscribe
+    end
+
+    def subscribe(q, &block)
+      MQ.queue(q).subscribe(:ack => true) do |info, data|
+        next if AMQP.closing?
+
+        args = decode(data)
+
+        if args['kill_by'] and args['kill_by'] < Time.now.to_i
+          info.ack
+          return
+        end
+
+        begin
+          results = block.call(args)
+        rescue RejectMessage
+          info.reject
+          next
+        rescue Object => e
+          puts "got exception #{e} - packing it for return"
+          ## just display the exception if there's not reply_to
+          results = { :exception => pack(e) } 
+        end
+
+        results = { :results => pack(results) } unless results.is_a?(Hash)
+        MQ.queue(args['reply_to']).publish(encode(results)) if args['reply_to']
+        info.ack
+      end
+      nil
+    end
+
+    def pack(obj)
+        [Marshal.dump(obj)].pack("m")
+    end
+
+    def unpack(s)
+        Marshal.load(s.unpack("m")[0])
+    end
+  end
+
+  class Binding
+    attr_accessor :key, :klass, :q, :subscribed, :active 
+    def initialize(key, klass, q, active)
+      @key = key
+      @klass = klass
+      @q = q
+      @subscribed = false
+      @active = active
+    end
+
+    def needs_to_subscribe?
+      handler = klass.new({}, {})
+      subscribed == false and handler.send(active) == true
+    end
+
+    def needs_to_unsubscribe?
+      handler = klass.new({}, {})
+      subscribed == true and handler.send(active) == false
+    end
+  end
+
+  module Routes
+    def bind_to(key, options = {})
+      key = key.to_s
+      q = "q.#{key}" if options[:route_to] == :one 
+      q = "q.#{ENV['HOSTNAME']}" if options[:route_to] == :host 
+      q = "q.#{rand 9_999_999}" if options[:route_to] == :all
+      raise "route_to => :one, :all or :host" unless q
+
+      active = options[:active]
+      active ||= :active
+
+      Routes.bindings[key] = Binding.new(key, self, q, active)
+    end
+
+    def self.check
+      bindings.each do |key, b|
+        if b.needs_to_subscribe?
+          Server.subscribe(b.q) do |params|
+            handler = b.klass.new(params, params)
+            handler.handle
+          end
+          b.subscribed = true
+        elsif b.needs_to_unsubscribe?
+          Server.unsubscribe(b.q)
+          b.subscribed = true
+        end
+      end
+    end
+
+    def self.bind
+      bindings.each do |key, b|
+        Server.bind(b.q, b.key)
+      end
+      check
+    end
+
+    def self.bindings
+      @bindings ||= {}
+    end
+  end
+
+  class Base < Vincent::Client
+    extend Routes
+
+    attr_accessor :headers, :params
+
+    def initialize(headers, params)
+      @headers = headers
+      @params = params
+    end
+
+    def [](index)
+      params[index]
+    end
+
+    def active
+      true
+    end
+
+    def reject
+      raise RejectMessage
+    end
+
+    def handle
+      if params['reply_to']
+        handle_call
+      else
+        handle_cast
+      end
+    end
+
+    def handle_call
+      raise MissingCallHandler
+    end
+
+    def handle_cast
+      raise MissingCastHandler
+    end
+
+  end
+end
+

data/lib/vincent/server.rb

@@ -0,0 +1,21 @@
+require 'vincent/base'
+
+module Vincent
+	class Server
+		extend Core
+
+		def self.start(&block)
+			EM.run { 
+				Signal.trap('INT') { AMQP.stop{ EM.stop } }
+				Signal.trap('TERM'){ AMQP.stop{ EM.stop } }
+
+				Fiber.new {
+					Vincent::Routes.bind
+					EM.add_periodic_timer(60) { Vincent::Routes.check }
+					block.call(Vincent::Server) if block
+				}.resume
+			}
+		end
+	end
+end
+

data/lib/vincent.rb

@@ -0,0 +1,50 @@
+
+require 'mq'
+require 'uri'
+require 'json'
+
+module Vincent
+
+	def self.method_missing(method,data = {})
+		key = method.to_s.gsub(/_/,".")
+		Vincent::Client.cast(key,data)
+	end
+
+	module Core
+    def exchange
+       @exchange ||= MQ.topic
+    end
+
+		def cast(key,args = {})
+			args['kill_by'] = Time.now.to_i + args['ttl'] if args['ttl'].to_i > 0
+			exchange.publish(encode(args), :routing_key => key)
+			nil
+		end
+
+		def encode(data)
+			data.to_json
+		end
+
+		def decode(data)
+			begin
+				JSON.parse(data)
+			rescue 
+        # log error
+				{}
+			end
+		end
+	end
+
+	class Client
+		extend Core
+	end
+end
+
+config = URI.parse(ENV['AMQP_URI'] || 'amqp://guest:guest@localhost/')
+AMQP.settings.merge!(
+  :host  => config.host,
+  :user  => config.user,
+  :pass  => config.password,
+  :vhost => config.path
+)
+

metadata

@@ -0,0 +1,57 @@
+--- !ruby/object:Gem::Specification 
+name: adamhenry-vincent
+version: !ruby/object:Gem::Version 
+  version: 0.0.2
+platform: ruby
+authors: 
+- Adam Henry
+- Orion Henry
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2009-03-20 00:00:00 -07:00
+default_executable: 
+dependencies: []
+
+description: An AMQP microp-framwork with best-practaces
+email: a.david.henry@gmail.com
+executables: []
+
+extensions: []
+
+extra_rdoc_files: []
+
+files: 
+- lib/vincent.rb
+- lib/vincent/server.rb
+- README
+- lib/vincent/base.rb
+has_rdoc: false
+homepage: http://www.vincent.com/
+post_install_message: 
+rdoc_options: []
+
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.2.0
+signing_key: 
+specification_version: 2
+summary: An AMQP micro-framwork
+test_files: []
+