dripdrop 0.0.3 → 0.1.0

data/.gitignore

@@ -12,6 +12,7 @@ tmtags
 
 ## VIM
 *.swp
+*.swo
 
 ## PROJECT::GENERAL
 coverage
@@ -19,4 +20,7 @@ rdoc
 doc
 pkg
 
+#RBX
+*.rbc
+
 ## PROJECT::SPECIFIC

data/README.md

@@ -1,115 +1,34 @@
-# dripdrop
+# DripDrop
 
-0MQ Based App Event Monitoring / processing.
-A work in progress.
+**Note: For now, for PUSH/PULL support you'll need to build zmqmachine from my fork of zmqmachine, the official gem does not yet have this**
 
-# Why use dripdrop?
+DripDrop is ZeroMQ(using zmqmachine) + Event Machine simplified for the general use case + serialization helpers.
 
-You want to asynchronously process / monitor arbitrary messages from your app.
-dripdrop does this well for a few reasons.
-
-* It's fast. dripdrop doesn't slow down your app. 0MQ + Bert are fast. Sending a message never blocks.
-* It's flexible. By leveraging 0MQ pub/sub sockets you can have many different processors (collectors in dripdrop) that don't impact or even care about each other
-* It's easy. Check out the agent and collector examples below. You can be processing stuff in no time.
-
-## An example with a WebSocket UI:
-
-### You'll need to have the zmq dev libs on your machine. On OSX this means
-
-1. Download and build zeromq from [zeromq.org](http://www.zeromq.org/area:download)
-1. The agent just uses the plain zmq gem, which runs fine on ruby 1.8.7+, *EDIT!* If you use my fork of ffi-rzmq everything should work with 1.8.7! This isn't fully tested yet though *END EDIT*  this is so you can use it in say your rails app. Everything else needs ruby 1.9.2 or jruby and uses Chuck Remes [ffi-rzmq](http://github.com/chuckremes/ffi-rzmq), and [zmqmachine](http://github.com/chuckremes/zmqmachine) gems which you must build yourself. I recommend using rvm to enable the use of multiple rubies on one machine.
-1. zmq_forwarder comes with zmq, use this to aggregate agent messages using the example config shown below
-
-### To run a simple example, feeding data to a websockets UI
-
-#### Aggregate agents with zmq_forwarder (comes with zmq)
-    $ zmq_forwarder examples/forwarder.cfg
-
-#### Start up the drip drop publisher example
-    $ drip-publisher
-
-#### Assuming you have mongodb running
-    $ drip-mlogger
-  
-#### Start up a webserver to host the HTML/JS for a sample websocket client
-    $ cd DRIPDROPFOLDER/example/web/
-    $ ruby server
-
-## Example Topology
-
-You can add as many listeners as you want, or reconfigure things any way you want. Heres how I plan on using it.
-
-![topology](http://github.com/andrewvc/dripdrop/raw/master/doc_img/topology.png "Topology")
-
-## Sending Messages
-
-Sending messages is easy with the agent, an example:
-
-    require 'rubygems'
-    require 'dripdrop/agent'
-
-    agent = DripDrop::Agent.new('tcp://127.0.0.1:2900')
-
-    loop do
-      #Test is the message name, this is the first part of the 0MQ message, used for filtering
-      #at the 0MQ sub socket level, :head is always a hash,  :body is freeform
-      #EVERYTHING must be serializable to BERT
-      agent.send_message('test', :body => 'hello', :head => {:key => 'value'})
-      puts "SEND"
-      sleep 1
-    end
-
-## Writing a custom message processor
-
-Writing custom message processors is super easy, just create a new DripDrop::Collector
-and run it. DripDrop::Collector is based on Chuck Remes' awesome zmqmachine, an evented
-0MQ processor. Heres' the MongoDB logger as an example:
-    
-    require 'rubygems'
-    require 'mongo'
-    require 'dripdrop/collector'
+Here's an example of the kind of thing DripDrop makes easy, from [examples/pubsub.rb](http://github.com/andrewvc/dripdrop/blob/master/example/pubsub.rb)
+ 
+    DripDrop::Node.new do |node|
+      z_addr = 'tcp://127.0.0.1:2200'
 
-    class DripDrop
-      class MLoggerCollector < Collector
-        attr_accessor :mongo_collection
-        
-        #Messages are a DripDrop::Message
-        def on_recv(message)
-          if @mongo_collection
-            @mongo_collection.insert(message.to_hash)
-          end
-        end
+      pub = node.zmq_publish(z_addr,:bind)
+      sub = node.zmq_subscribe(z_addr,:connect).on_recv do |message|
+        puts "Receiver 1 #{message.inspect}"
+      end
+      sub = node.zmq_subscribe(z_addr, :connect).on_recv do |message|
+        puts "Receiver 2 #{message.inspect}"
       end
 
-      class MLogger
-        attr_reader :sub_address, :sub_reactor, :mongo_host, :mongo_port, :mongo_db,
-                    :mongo_connection, :mongo_collection
-
-        def initialize(sub_address='tcp://127.0.0.1:2901',mhost='127.0.0.1',mport=27017,mdb='dripdrop')
-          @sub_address   = URI.parse(sub_address)
-          @sub_collector = MLoggerCollector.new('tcp://127.0.0.1:2901')
-          
-          @mongo_host, @mongo_port, @mongo_db = mhost, mport, mdb
-          @mongo_connection = Mongo::Connection.new(@mongo_host,@mongo_port).db(@mongo_db)
-          @mongo_collection = @mongo_connection.collection('raw')
-        end
-
-        def run
-          @sub_collector.mongo_collection = @mongo_collection
-          @sub_collector.run.join
-        end
+      node.zm_reactor.periodical_timer(5) do
+        pub.send_message(DripDrop::Message.new('test', :body => 'Test Payload'))
       end
     end
+   
 
+Want to see a longer example encapsulating both zmqmachine and eventmachine functionality? Check out [this file](http://github.com/andrewvc/dripdrop-webstats/blob/master/lib/dripdrop-webstats.rb), which encapsulates all the functionality of the diagram below:
 
-## Note on Patches/Pull Requests
- 
-* Fork the project.
-* Make your feature addition or bug fix.
-* Commit, do not mess with rakefile, version, or history.
-  (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
-* Send me a pull request. Bonus points for topic branches.
+![topology](http://github.com/andrewvc/dripdrop/raw/master/doc_img/topology.png "Topology")
+
+#How It Works
 
-## Copyright
+DripDrop encapsulates both zmqmachine, and eventmachine. It provides some sane default messaging choices, using [BERT](http://github.com/blog/531-introducing-bert-and-bert-rpc) (A binary, JSON, like serialization format) and JSON to automatically. zmqmachine and eventmachine have some good APIs, some convoluted ones, the goal here is to smooth over the bumps, and make writing highly concurrent programs both as terse and beautiful as possible.
 
 Copyright (c) 2010 Andrew Cholakian. See LICENSE for details.

data/Rakefile

@@ -10,8 +10,10 @@ begin
     gem.email = "andrew@andrewvc.com"
     gem.homepage = "http://github.com/andrewvc/dripdrop"
     gem.authors = ["Andrew Cholakian"]
-    gem.add_dependency('zmq')
-    gem.add_dependency('ffi')
+    gem.add_dependency('ffi-rzmq')
+    gem.add_dependency('eventmachine')
+    gem.add_dependency('bert')
+    gem.add_dependency('json')
   end
   Jeweler::GemcutterTasks.new
 rescue LoadError

data/VERSION

@@ -1 +1 @@
-0.0.3
+0.1.0

data/dripdrop.gemspec

@@ -5,14 +5,13 @@
 
 Gem::Specification.new do |s|
   s.name = %q{dripdrop}
-  s.version = "0.0.3"
+  s.version = "0.1.0"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Andrew Cholakian"]
-  s.date = %q{2010-08-14}
+  s.date = %q{2010-09-05}
   s.description = %q{0MQ App stats}
   s.email = %q{andrew@andrewvc.com}
-  s.executables = ["drip-mlogger", "drip-publisher"]
   s.extra_rdoc_files = [
     "LICENSE",
      "README.md"
@@ -24,23 +23,17 @@ Gem::Specification.new do |s|
      "README.md",
      "Rakefile",
      "VERSION",
-     "bin/drip-mlogger",
-     "bin/drip-publisher",
      "doc_img/topology.png",
      "dripdrop.gemspec",
      "example/agent_test.rb",
-     "example/forwarder.cfg",
-     "example/web/public/js/jquery.gracefulWebSocket.js",
-     "example/web/public/js/jquery.js",
-     "example/web/public/js/jquery.websocket.js",
-     "example/web/public/view.html",
-     "example/web/server.rb",
+     "example/pubsub.rb",
+     "example/pushpull.rb",
      "lib/dripdrop.rb",
      "lib/dripdrop/agent.rb",
-     "lib/dripdrop/collector.rb",
+     "lib/dripdrop/handlers/websockets.rb",
+     "lib/dripdrop/handlers/zeromq.rb",
      "lib/dripdrop/message.rb",
-     "lib/dripdrop/mlogger.rb",
-     "lib/dripdrop/publisher.rb"
+     "lib/dripdrop/node.rb"
   ]
   s.homepage = %q{http://github.com/andrewvc/dripdrop}
   s.rdoc_options = ["--charset=UTF-8"]
@@ -53,15 +46,21 @@ Gem::Specification.new do |s|
     s.specification_version = 3
 
     if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
-      s.add_runtime_dependency(%q<zmq>, [">= 0"])
-      s.add_runtime_dependency(%q<ffi>, [">= 0"])
+      s.add_runtime_dependency(%q<ffi-rzmq>, [">= 0"])
+      s.add_runtime_dependency(%q<eventmachine>, [">= 0"])
+      s.add_runtime_dependency(%q<bert>, [">= 0"])
+      s.add_runtime_dependency(%q<json>, [">= 0"])
     else
-      s.add_dependency(%q<zmq>, [">= 0"])
-      s.add_dependency(%q<ffi>, [">= 0"])
+      s.add_dependency(%q<ffi-rzmq>, [">= 0"])
+      s.add_dependency(%q<eventmachine>, [">= 0"])
+      s.add_dependency(%q<bert>, [">= 0"])
+      s.add_dependency(%q<json>, [">= 0"])
     end
   else
-    s.add_dependency(%q<zmq>, [">= 0"])
-    s.add_dependency(%q<ffi>, [">= 0"])
+    s.add_dependency(%q<ffi-rzmq>, [">= 0"])
+    s.add_dependency(%q<eventmachine>, [">= 0"])
+    s.add_dependency(%q<bert>, [">= 0"])
+    s.add_dependency(%q<json>, [">= 0"])
   end
 end
 

data/example/agent_test.rb

@@ -1,7 +1,7 @@
 require 'rubygems'
 require 'dripdrop/agent'
 
-agent = DripDrop::Agent.new('tcp://127.0.0.1:2900')
+agent = DripDrop::Agent.new(ZMQ::PUB,'tcp://127.0.0.1:2900',:connect)
 
 loop do
   #Test is the message name, this is the first part of the 0MQ message, used for filtering

data/example/pubsub.rb

@@ -0,0 +1,18 @@
+require 'dripdrop/node'
+Thread.abort_on_exception = true
+
+DripDrop::Node.new do |node|
+  z_addr = 'tcp://127.0.0.1:2200'
+    
+  pub = node.zmq_publish(z_addr,:bind)
+  sub = node.zmq_subscribe(z_addr,:connect).on_recv do |message|
+    puts "Receiver 1 #{message.inspect}"
+  end
+  sub = node.zmq_subscribe(z_addr, :connect).on_recv do |message|
+    puts "Receiver 2 #{message.inspect}"
+  end
+  
+  node.zm_reactor.periodical_timer(5) do
+    pub.send_message(DripDrop::Message.new('test', :body => 'Test Payload'))
+  end
+end

data/example/pushpull.rb

@@ -0,0 +1,21 @@
+require 'dripdrop/node'
+Thread.abort_on_exception = true
+
+DripDrop::Node.new do |node|
+  z_addr = 'tcp://127.0.0.1:2200'
+   
+  node.zmq_pull(z_addr, :connect).on_recv do |message|
+    puts "Receiver 2 #{message.body}"
+  end
+  node.zmq_pull(z_addr, :connect).on_recv do |message|
+    puts "Receiver 1 #{message.body}"
+  end
+  push = node.zmq_push(z_addr, :bind)
+
+  i = 0
+  node.zm_reactor.periodical_timer(800) do
+    i += 1
+    puts i
+    push.send_message(DripDrop::Message.new('test', :body => "Test Payload #{i}"))
+  end
+end

data/lib/dripdrop/agent.rb

@@ -1,5 +1,13 @@
 require 'dripdrop/message'
-require 'zmq'
+#Check if we're in 1.8.7
+unless defined?(RUBY_ENGINE)
+  require 'zmq'
+  ZMQGEM = :rbzmq
+else
+  require 'ffi-rzmq'
+  ZMQGEM = :ffirzmq
+end
+require 'uri'
 require 'bert'
 
 class DripDrop
@@ -8,16 +16,26 @@ class DripDrop
     attr_reader :address, :context, :socket
     
     #address should be a string like tcp://127.0.0.1
-    def initialize(address)
-      @address = address
+    def initialize(sock_type,address,sock_ctype)
       @context = ZMQ::Context.new(1)
-      @socket  = @context.socket(ZMQ::PUB)
-      @socket.connect(@address)
+      @socket  = @context.socket(sock_type)
+      if sock_ctype == :bind
+        @socket.bind(address)
+      else
+        @socket.connect(address)
+      end
     end
 
     #Sends a DripDrop::Message to the socket
     def send_message(name,body,head={})
-      puts @socket.send(DripDrop::Message.new(name,:body => body, :head => head).encoded, 0)
+      message = DripDrop::Message.new(name,:body => body, :head => head).encoded
+      if ZMQGEM == :rbzmq
+        @socket.send name, ZMQ::SNDMORE
+        @socket.send message
+      else
+        @socket.send_string name, ZMQ::SNDMORE
+        @socket.send_string message
+      end
     end
   end
 end

data/lib/dripdrop/handlers/websockets.rb

@@ -0,0 +1,68 @@
+require 'em-websocket'
+require 'json'
+
+class DripDrop
+  class WebSocketHandler
+    attr_reader :ws, :address, :thread
+   
+    def initialize(address,opts={})
+      @raw    = false #Deal in strings or ZMQ::Message objects
+      @thread = Thread.new do
+          host, port = address.host, address.port.to_i
+          @debug = opts[:debug] || false
+
+          ws_conn = EventMachine::WebSocket::Connection
+          EventMachine::start_server(host,port,ws_conn,:debug => @debug) do |ws|
+            @ws = ws
+            @ws.onopen do
+              @onopen_handler.call(ws) if @onopen_handler
+            end
+            @ws.onmessage do |message|
+              unless @raw
+                begin
+                  parsed = JSON.parse(message)
+                  message = DripDrop::Message.new(parsed['name'], :body => parsed['body'], :head => parsed['head'] || {})
+                rescue StandardError => e
+                  puts "Could not parse message: #{e.message}"
+                end
+              end
+              @onmessage_handler.call(message,ws) if @onmessage_handler
+            end
+            @ws.onclose do
+              @onclose_handler.call(@ws) if @onclose_handler
+            end
+            @ws.onerror do
+              @onerror_handler.call(@ws) if @onerror_handler
+            end
+          end
+      end
+    end
+    
+    def on_recv(&block)
+      @raw = false
+      @onmessage_handler = block
+      self
+    end
+
+    def on_recv_raw(&block)
+      @raw = true
+      @onmessage_handler = block
+      self
+    end
+
+    def on_open(&block)
+      @onopen_handler = block
+      self
+    end
+  
+    def on_close(&block)
+      @onclose_handler = block
+      self
+    end
+    
+    def on_error(&block)
+      @onerror_handler = block
+      self
+    end
+  end
+end

data/lib/dripdrop/handlers/zeromq.rb

@@ -0,0 +1,136 @@
+require 'ffi-rzmq'
+
+class DripDrop
+  class ZMQBaseHandler
+    attr_reader :address, :socket_ctype, :socket
+    
+    def initialize(address,zm_reactor,socket_ctype,opts={})
+      @address      = address
+      @zm_reactor   = zm_reactor
+      @socket_ctype = socket_ctype # :bind or :connect
+      @debug        = opts[:debug]
+    end
+    
+    def on_attach(socket)
+      @socket = socket
+      if    @socket_ctype == :bind
+        socket.bind(@address)
+      elsif @socket_ctype == :connect
+        socket.connect(@address)
+      else
+        raise "Unsupported socket ctype '#{@socket_ctype}'"
+      end
+    end
+     
+    def on_recv(msg_format=:dripdrop,&block)
+      @msg_format = msg_format 
+      @recv_cbak = block
+      self
+    end
+  end
+
+  class ZMQWritableHandler < ZMQBaseHandler
+    def initialize(*args)
+      super(*args)
+      @send_queue = []
+    end
+  end
+  
+  class ZMQReadableHandler < ZMQBaseHandler
+    def initialize(*args,&block)
+      super(*args)
+      @recv_cbak = block
+    end
+  end
+  
+  class ZMQSubHandler < ZMQReadableHandler
+    attr_reader :address, :socket_ctype
+    
+    def on_attach(socket)
+      super(socket)
+      socket.subscribe('')
+    end
+
+    def on_readable(socket, messages)
+      if    @msg_format == :raw
+        @recv_cbak.call(messages)
+      elsif @msg_format == :dripdrop
+        unless messages.length == 2
+          puts "Expected pub/sub message to come in two parts" 
+          return false
+        end
+        topic = messages.shift.copy_out_string
+        body  = messages.shift.copy_out_string
+        msg   = @recv_cbak.call(DripDrop::Message.decode(body))
+      else
+        raise "Unsupported message format '#{@msg_format}'"
+      end
+    end
+  end
+    
+  
+  class ZMQPubHandler < ZMQWritableHandler
+    #Send any messages buffered in @send_queue
+    def on_writable(socket)
+      unless @send_queue.empty?
+        message = @send_queue.shift
+        
+        num_parts = message.length
+        message.each_with_index do |part,i|
+          multipart = i + 1 < num_parts ? true : false
+          if part.class == ZMQ::Message
+            socket.send_message(part, multipart)
+          else
+            socket.send_message_string(part, multipart)
+          end
+        end
+      else
+        @zm_reactor.deregister_writable(socket)
+      end
+    end
+    
+    #Sends a message along
+    def send_message(message)
+      if message.is_a?(DripDrop::Message)
+        @send_queue.push([message.name, message.encoded])
+      elsif message.is_a?(Array)
+        @send_queue.push(message)
+      else
+        @send_queue.push([message])
+      end
+      @zm_reactor.register_writable(@socket)
+    end
+  end
+
+  class ZMQPullHandler < ZMQReadableHandler
+    def on_readable(socket, messages)
+      if @msg_format == :raw
+        @recv_cbak.call(messages)
+      else
+        body  = messages.shift.copy_out_string
+        msg   = @recv_cbak.call(DripDrop::Message.decode(body))
+      end
+    end
+  end
+
+  class ZMQPushHandler < ZMQWritableHandler
+    def on_writable(socket)
+      unless @send_queue.empty?
+        message = @send_queue.shift
+        socket.send_message_string(message)
+      else
+        @zm_reactor.deregister_writable(socket)
+      end
+    end
+    
+    #Sends a message along
+    def send_message(message)
+      if message.is_a?(DripDrop::Message)
+        @send_queue.push(message.encoded)
+        @zm_reactor.register_writable(@socket)
+      else
+        @send_queue.push(message)
+      end
+    end
+  end
+end

data/lib/dripdrop/message.rb

@@ -3,12 +3,14 @@ require 'bert'
 
 class DripDrop
   #DripDrop::Message messages are exchanged between all tiers in the architecture
-  #A Message is composed of a name, head, and body. The name exists primarily for the
-  #purpose of native ZMQ filtering, since ZMQ can filter based on a message prefix.
+  #A Message is composed of a name, head, and body, and should be restricted to types that
+  #can be readily encoded to JSON, that means hashes, arrays, strings, integers, and floats
+  #internally, they're just stored as BERT
   #
-  #The name is any string consisting of non-null chars.
-  #The rest of the payload is a BERT encoded head and body, both of which are hashes.
-  #The head and body don't have rigid definitions yet, use as you please.
+  #The basic message format is built to mimic HTTP. Why? Because I'm a dumb web developer :)
+  #The name is kind of like the URL, its what kind of message this is.
+  #head should be used for metadata, body for the actual data.
+  #These definitions are intentionally loose, because protocols tend to be used loosely.
   class Message
     attr_accessor :name, :head, :body
     
@@ -28,7 +30,7 @@ class DripDrop
     
     #The encoded message, ready to be sent across the wire via ZMQ
     def encoded
-      "#{@name}\0#{BERT.encode({:head => @head, :body => @body})}"
+      BERT.encode(self.to_hash)
     end
     
     #Convert the Message to a hash like:
@@ -37,11 +39,25 @@ class DripDrop
       {:name => @name, :head => @head, :body => @body}
     end
 
-    #Parses an encoded message
+    #Parses an already encoded string
+    def self.decode(*args); self.parse(*args) end
     def self.parse(msg)
-      name, encoded_body = msg.split("\0",2)
-      decoded = BERT.decode(encoded_body)
-      self.new(name, :head => decoded[:head], :body => decoded[:body])
+      return nil if msg.nil? || msg.empty?
+      #This makes parsing ZMQ messages less painful, even if its ugly here
+      #We check the class name as a string if case we don't have ZMQ loaded
+      if msg.class.to_s == 'ZMQ::Message'
+        msg = msg.copy_out_string 
+        return nil if msg.empty?
+      end
+      decoded = BERT.decode(msg)
+      self.new(decoded[:name], :head => decoded[:head], :body => decoded[:body])
+    end
+    
+    private
+    
+    #Sanitize a string so it'll look good for JSON, BERT, and MongoDB
+    def sanitize_structure(structure)
+      #TODO: Make this work, and called for head, and body
     end
   end
 end