em-zmq-tp10 0.1.7

data/.gitignore

@@ -0,0 +1,18 @@
+*.swp
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in em-zmq-tp10.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2012 Sokolov Yura 'funny-falcon'
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,237 @@
+# EventMachine ZMTP1.0 protocol (ZMQ2.x protocol)
+
+It is implementation of ZMTP 1.0 - ZMQ 2.x transport protocol using facilites provided by EventMachine.
+There are implementations of ZMQ socket types which try to be similar to original, but not too hard. Moreover, you may create your own behaviour.
+Library is tested against native ZMQ using ffi-rzmq.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'em-zmq-tp10'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install em-zmq-tp10
+
+## Usage
+
+Library provides callback oriented classes which tries to emulate behaviour of standard ZMQ classes, but already integrated with EventMachine eventloop.
+
+Main difference in behaviour is in highwatermark handling and balancing:
+For DEALER and REQ zmq provides roundrobin load balancing until HighWaterMark reached, then send operation blocks or returns error `EAGAIN`.
+This implementation do roundrobin until fixed internal highwatermark (2048bytes) reached, and then pushes to common queue until userdefined watermark is reached.
+
+This internal per connect buffer is handled by EventMachine itself, and there is no precise control over it, so that, if peer is disconnected, all pushed messages to this buffer are lost. So that, compared to raw ZMQ, you loose not only content of OS's internal socket buffer, but EventMachine buffer as well :(
+
+But since ZMQ is never pretended on durability, it is not big issue (for me).
+
+There is two strategy of highwatermark handling: `drop_first` and `drop_last`.
+`drop_last` - is ignoring any try to send message if queue is full - this is default strategy for ZMQ if you use nonblocking sending.
+`drop_first` - dropping earliest message in a queue, so that newly inserted message will have more chanches to be sent. You can react on such dropping by overriding `cancel_message` (or `cancel_request` for Req). I like this strategy more cause old request tends to be less usefull, but `drop_last` is still default for "compatibility".
+
+There is also simplified classes without internal queue (`PreDealer`, `PreReq`, `PreRouter`, `PreRep`, `PrePub`), so that you can implement your own strategy of queueing.
+
+And you could do any crazy thing using base `EM::Protocols::Zmq2::Socket` class
+
+
+### Socket
+
+  Base class. It provides #connect and #bind methods for establishing endpoints.
+  This method could be called outside EM event loop (even before EM.run called), cause they use EM.schedule.
+  TCP and IPC endpoints are supported and fully interoperable with native ZMQ.
+  INPROC supported as well, but you should treat them as connections inside EventMachine's context, so that you could not connect to native ZMQ inproc endpoints.
+
+### Dealer
+
+      class MyPreDealer < EM::Protocols::Zmq2::PreDealer
+        def receive_message(message)
+          puts "Message received: #{message.inspect}"
+        end
+      end
+      dealer = MyPreDealer.new
+      dealer.connect('tcp://127.0.0.1:8000')
+      dealer.bind('unix://dealer.sock')
+      EM.schedule {
+        if !dealer.send_message(['asdf','fdas'])
+          puts "Could not send message (no free peers)"
+        end
+      }
+
+      class MyDealer < EM::Protocols::Zmq2::Dealer
+        def receive_message(message)
+          puts "Message received: #{message.inspect}"
+        end
+      end
+      dealer = MyDealer.new(hwm: 1000, hwm_strategy: :drop_last)
+      dealer.connect('tcp://127.0.0.1:8000')
+      EM.schedule {
+        if !dealer.send_message(['asdf','fdas'])
+          puts "No free peers and outgoing queue is full"
+        end
+      }
+
+      dealer = EM::Protocols::Zmq2::DealerCb.new do |message|
+         puts "Receive message #{message.inspect}"
+      end
+      dealer.connect('ipc://rep')
+      EM.schedule {
+        dealer.send_message(['hello','world'])
+      }
+
+### Req
+
+      class MyPreReq < EM::Protocols::Zmq2::PreReq
+        def receive_reply(message, data, request_id)
+          puts "Received message #{message} and stored data #{data}
+        end
+      end
+      req = MyPreReq.new
+      req.bind(...)
+      req.connect(...)
+      if request_id = req.send_request(['this is', 'message'], 'saved_data')
+        puts "Message sent"
+      else
+        puts "there is no free peer"
+      end
+
+      class MyReq < EM::Protocols::Zmq2::PreReq
+        def receive_reply(message, data, request_id)
+          puts "Received message #{message} and stored data #{data}
+        end
+      end
+      req = MyReq.new
+      req.bind(...)
+      req.connect(...)
+      if request_id = req.send_request(['hi'], 'ho')
+        puts "Message sent"
+      end
+
+      req = EM::Protocols::Zmq2::ReqCb.new
+      req.bind('ipc://req')
+      timer = nil
+      request_id = req.send_request(['hello', 'world']) do |message|
+        EM.cancel_timer(timer)
+        puts "Message #{message}"
+      end
+      if request_id
+        timer = EM.add_timer(1) {
+          req.cancel_request(request_id)
+        }
+      end
+
+      req = EM::Protocols::Zmq2::ReqDefer.new
+      req.bind('ipc://req')
+      data = {hi: 'ho'}
+      deferable = req.send_request(['hello', 'world'], data) do |reply, data|
+        puts "Reply received #{reply} #{data}"
+      end
+      deferable.timeout 1
+      deferable.errback do
+        puts "Message canceled"
+      end
+      deferable.callback do |reply, data|
+        puts "Another callback #{reply} #{data}"
+      end
+
+### Router
+
+Router stores peer identity in a message, as ZMQ router do.
+And it sends message to a peer, which idenitity equals to first message string.
+`PreRouter` does no any queue caching, `Router` saves message in queue per peer, controlled by highwatermark strategy.
+
+      class MyPreRouter < EM::Protocols::Zmq2::PreRouter
+        def receive_message(message)
+          puts "Received message #{message} (and it contains envelope)"
+        end
+      end
+      router = MyPreRouter.new
+      router.bind(...)
+      router.send_message(message)
+
+      class MyRouter < EM::Protocols::Zmq2::Router
+        def receive_message(message)
+          puts "Received message #{message}"
+          message[-1] = 'reply'
+          send_message(message)
+        end
+      end
+      router = MyPreRouter.new(hwm: 1000, hwm_strategy: :drop_first)
+      router.bind(...)
+      router.send_message(message)
+
+### Rep
+
+REP differs from Router mainly in methods signature.
+
+      class EchoBangPreRep < EM::Protocols::Zmq2::PreRep
+        def receive_request(message, envelope)
+          message << "!"
+          if send_reply(message, envelope)
+            puts "reply sent successfuly"
+          end
+        end
+      end
+      rep = EchoBangPreRep.new
+      rep.bind('ipc://rep')
+
+      class EchoBangRep < EM::Protocols::Zmq2::Rep
+        def receive_request(message, envelope)
+          message << "!"
+          if send_reply(message, envelope)
+            puts "reply sent successfuly"
+          end
+        end
+      end
+      rep = EchoBangRep.new
+      rep.bind('ipc://rep')
+
+### Sub
+
+Unless ZMQ sub, this `Sub` accepts not only strings, but also RegExps and procs for subscribing.
+Note that as in ZMQ 2.x filtering occurs on Sub side.
+
+Since subscriptions could be defined with callback passed to `:subscribe` option, `subscribe` or `subscribe_many` methods, you could use this class without overloading.
+
+      class MySub < EM::Protocols::Zmq2::Sub
+        def receive_message(message)
+          puts "default handler"
+        end
+      end
+      sub = MySub.new(subscribe: ['this', 'that'])
+      sub.subscribe /^callback/i, do |message|
+        puts "Callback subscribe #{message}"
+      end
+      sub.subscribe_many(
+        proc{|s| s.end_with?("END")} => proc{|message| puts "TILL END #{message}"},
+        '' => nil # also to default
+      )
+
+### Pub
+
+`PrePub` sends messages only to connected and not busy peers. `send_message` returns true, if there is at least one peer with short EventMachine's outgoing queue, to which message is scheduled.
+
+`Pub` tries to queue messages for all connected and for disconnected peers with explicit identity set.
+
+Since there is no incoming data, there is no need to overload methods.
+
+      pub = EM::Protocols::Zmq2::PrePub.new
+      pub.bind(...)
+      pub.send_message(['hi', 'you'])
+
+      pub = EM::Protocols::Zmq2::Pub.new
+      pub.bind(...)
+      pub.send_message(['hi', 'you'])
+
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Added some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1,9 @@
+#!/usr/bin/env rake
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+Rake::TestTask.new do |t|
+  t.libs.push "lib"
+  t.test_files = FileList['tests/test_*.rb']
+  t.verbose = true
+end

data/em-zmq-tp10.gemspec

@@ -0,0 +1,23 @@
+# -*- encoding: utf-8 -*-
+require File.expand_path('../lib/em/protocols/zmq2/version', __FILE__)
+
+Gem::Specification.new do |gem|
+  gem.authors       = ["Sokolov Yura 'funny-falcon'"]
+  gem.email         = ["funny.falcon@gmail.com"]
+  gem.description   = %q{Implementation of ZMQ2.x transport protocol - ZMTP1.0}
+  gem.summary       = %q{
+Implementation of ZMQ2.1 transport protocol and main socket types using EventMachine
+for managing TCP/UNIX connections, doing nonblocking writes and calling callback on
+incoming messages.}
+  gem.homepage      = "https://github.com/funny-falcon/em-zmq-tp10"
+
+  gem.files         = `git ls-files`.split($\)
+  gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
+  gem.name          = "em-zmq-tp10"
+  gem.require_paths = ["lib"]
+  gem.version       = EventMachine::Protocols::Zmq2::VERSION
+  gem.add_runtime_dependency 'eventmachine'
+  gem.add_development_dependency 'ffi-rzmq'
+  gem.required_ruby_version = '>= 1.9.2'
+end

data/lib/em-zmq-tp10.rb

@@ -0,0 +1 @@
+require "em/protocols/zmq2"

data/lib/em/protocols/zmq2.rb

@@ -0,0 +1,25 @@
+module EventMachine
+  module Protocols
+    module Zmq2
+      EMPTY = ''.freeze
+      SMALL_TIMEOUT = 0.03
+      HWM_INFINITY = 2**20
+      autoload :PreDealer, 'em/protocols/zmq2/dealer.rb'
+      autoload :Dealer, 'em/protocols/zmq2/dealer.rb'
+      autoload :DealerCb, 'em/protocols/zmq2/dealer.rb'
+      autoload :PreReq, 'em/protocols/zmq2/req.rb'
+      autoload :Req, 'em/protocols/zmq2/req.rb'
+      autoload :ReqCb, 'em/protocols/zmq2/req.rb'
+      autoload :ReqDefer, 'em/protocols/zmq2/req.rb'
+      autoload :PreRouter, 'em/protocols/zmq2/router.rb'
+      autoload :Router, 'em/protocols/zmq2/router.rb'
+      autoload :PreRep, 'em/protocols/zmq2/rep.rb'
+      autoload :Rep, 'em/protocols/zmq2/rep.rb'
+      autoload :Sub, 'em/protocols/zmq2/pub_sub.rb'
+      autoload :PrePub, 'em/protocols/zmq2/pub_sub.rb'
+      autoload :Pub, 'em/protocols/zmq2/pub_sub.rb'
+    end
+  end
+end
+require 'em/protocols/zmq2/socket_connection'
+require 'em/protocols/zmq2/socket'

data/lib/em/protocols/zmq2/connection.rb

@@ -0,0 +1,37 @@
+module EventMachine
+  module Protocols
+    module Zmq2
+      module ConnectionMixin
+        def sent_data
+          @socket.peer_free(@peer_identity, self) if not_too_busy?
+        end
+
+        def not_too_busy?
+          free = _not_too_busy?
+          self.notify_when_free = !free
+          free
+        end
+
+        def post_init
+          send_strings @socket.identity
+        end
+
+        def receive_strings(message)
+          unless @peer_identity
+            peer_identity = message.first
+            @peer_identity = @socket.register_peer(peer_identity, self)
+          else
+            @socket.receive_message_and_peer message, @peer_identity
+          end
+        end
+
+        def unbind
+          if @peer_identity
+            @socket.unregister_peer(@peer_identity)
+          end
+          @socket.not_connected(self)
+        end
+      end
+    end
+  end
+end

data/lib/em/protocols/zmq2/dealer.rb

@@ -0,0 +1,133 @@
+require 'em/protocols/zmq2/socket'
+module EventMachine
+  module Protocols
+    module Zmq2
+      # ZMQ socket which acts like a Dealer, but without any message queueing
+      # (except for EventMachine write buffers). So that, it is up to you
+      # to react when message could not be send
+      #
+      # @example
+      # class MyDealer < EM::Protocols::Zmq2::PreDealer
+      #   def receive_message(message)
+      #     puts "Message received: #{message.inspect}"
+      #   end
+      # end
+      # dealer = MyDealer.new
+      # dealer.connect('tcp://127.0.0.1:8000')
+      # if !dealer.send_message(['asdf','fdas'])
+      #   puts "Could not send message (no free peers)"
+      # end
+      class PreDealer < Socket
+        # @private
+        def choose_peer(even_if_busy = false)
+          peers = even_if_busy ? @peers : @free_peers
+          i = peers.size
+          while i > 0
+            ident, connect = peers.shift
+            if even_if_busy || connect.not_too_busy?
+              peers[ident] = connect # use the fact, that hash is ordered in Ruby 1.9
+              return connect  unless connect.error?
+            end
+            i -= 1
+          end
+        end
+
+        # overrides default callback to call +#receive_message+
+        def receive_message_and_peer(message, peer_identity) # :nodoc:
+          receive_message(message)
+        end
+
+        # override to have desired reaction on message
+        def receive_message(message)
+          raise NoMethodError
+        end
+
+        # tries to send message.
+        # @param [Array] message - message to be sent (Array of Strings or String)
+        # @param [Boolean] even_if_busy - ignore busyness of connections
+        # @return [Boolean] whether message were sent
+        def send_message(message, even_if_busy = false)
+          if connect = choose_peer(even_if_busy)
+            connect.send_strings(message)
+            true
+          end
+        end
+      end
+
+      # ZMQ socket which tries to be a lot like a Dealer. It stores messages into outgoing
+      # queue, it tries to balance on socket busyness (which is slightely more, than ZMQ do).
+      # The only visible change from PreDealer is less frequent +send_message+ false return.
+      #
+      # @example
+      # class MyDealer < EM::Protocols::Zmq2::Dealer
+      #   def receive_message(message)
+      #     puts "Message received: #{message.inspect}"
+      #   end
+      # end
+      # dealer = MyDealer.new
+      # dealer.connect('tcp://127.0.0.1:8000')
+      # if !dealer.send_message(['asdf','fdas'])
+      #   puts "No free peers and outgoing queue is full"
+      # end
+      class Dealer < PreDealer
+        # :stopdoc:
+        def initialize(opts = {})
+          super
+          @write_queue = []
+        end
+
+        alias raw_send_message send_message
+        def flush_queue(even_if_busy = false)
+          until @write_queue.empty?
+            return false  unless raw_send_message(@write_queue.first, even_if_busy)
+            @write_queue.shift
+          end
+          true
+        end
+
+        def send_message(message)
+          flush_queue && super(message) || push_to_queue(@write_queue, message)
+        end
+
+        def peer_free(peer_identity, connection)
+          super
+          flush_queue
+        end
+
+      private
+        def flush_all_queue
+          flush_queue(true)
+        end
+
+        def react_on_hwm_decrease
+          push_to_queue(@write_queue)
+        end
+        # :startdoc:
+      end
+
+      # Convinient Dealer class which accepts callback in constructor, which will be called
+      # on every incoming message (instead of #receive_message)
+      #
+      # @example
+      # dealer = EM::Protocols::Zmq2::DealerCb.new do |message|
+      #   puts "Receive message #{message.inspect}"
+      # end
+      # dealer.connect('ipc://rep')
+      # dealer.send_message(['hello','world'])
+      class DealerCb < Dealer
+        # Accepts callback as second parameter or block
+        # :callsec:
+        #   new(opts) do |message|   end
+        #   new(opts, proc{|message| })
+        def initialize(opts = {}, cb = nil, &block)
+          super opts
+          @read_callback = cb || block
+        end
+
+        def receive_message(message) # :nodoc:
+          @read_callback.call message
+        end
+      end
+    end
+  end
+end