arachni-rpc-em 0.1.3 → 0.2

data/CHANGELOG.md

@@ -1,9 +1,25 @@
 # ChangeLog
 
+## Version 0.2 _(June 23, 2013)_
+
+- YAML engine no longer forced to _Syck_.
+- Added support for UNIX domain sockets.
+- `Client`
+    - Moved connection handler to its own class file.
+    - Updated to reuse connections whenever possible.
+    - Maintains an adjustable-sized connection pool.
+        - Uses a single connection by default.
+- `Server`
+    - Moved connection handler to its own class file.
+    - Removed connection inactivity timeout.
+- Cleaned up RSpec tests.
+- Added Bundler files.
+
 ## Version 0.1.3 _(April 15, 2013)_
 
 - Stopped client callbacks from being deferred.
-- Server now supports a fallback serializer to allow clients to use a secondary serializer if they so choose.
+- Server now supports a fallback serializer to allow clients to use a secondary
+    serializer if they so choose.
 - Client request-retry strategy tweaked to be more resilient.
 
 ## Version 0.1.2

data/README.md

@@ -3,7 +3,7 @@
 <table>
     <tr>
         <th>Version</th>
-        <td>0.1.3</td>
+        <td>0.2</td>
     </tr>
     <tr>
         <th>Github page</th>
@@ -33,25 +33,33 @@
 
 ## Synopsis
 
-Arachni-RPC EM is an implementation of the <a href="http://github.com/Arachni/arachni-rpc">Arachni-RPC</a> protocol using EventMachine and provides both a server and a client. <br/>
+Arachni-RPC EM is an implementation of the <a href="http://github.com/Arachni/arachni-rpc">Arachni-RPC</a>
+protocol using EventMachine and provides both a server and a client. <br/>
 
 ## Features
 
 It's capable of:
 
- - Performing and handling a few thousand requests per second (depending on call size, network conditions and the like).
- - Configurable retry-on-fail for requests.
- - TLS encryption (with peer verification).
- - Asynchronous and synchronous requests.
- - Handling server-side asynchronous calls that require a block (or any method that passes its result to a block instead of returning it).
- - Token-based authentication.
- - Primary and secondary (fallback) serializers -- Server will expect the Client to use the primary serializer,
-    if the Request cannot be parsed using the primary one, it will revert to using the fallback to parse the Request and serialize the Response.
+- Performing and handling a few thousand requests per second (depending on call
+    size, network conditions and the like).
+- Operating over TCP/IP and UNIX domain sockets.
+- Configurable retry-on-failure for requests.
+- Keep-alive and connection re-use.
+- TLS encryption (with peer verification).
+- Asynchronous and synchronous requests.
+- Handling server-side asynchronous calls that require a block (or any method
+    that passes its result to a block instead of returning it).
+- Token-based authentication.
+- Primary and secondary/fallback serializers
+    - Server will expect the Client to use the primary serializer, if the Request
+        cannot be parsed using the primary one, it will revert to using the
+        fallback to parse the Request and serialize the Response.
 
 ## Usage
 
-Check out the files in the <i>examples/</i> directory, they go through everything in great detail.<br/>
-The tests under <i>spec/arachni/rpc/</i> cover everything too so they can probably help you out.
+The files in the `examples/` directory go through everything in great detail.
+Also, the tests under `spec/arachni/rpc/` cover everything too so they can
+provide you with hints.
 
 ## Installation
 
@@ -65,15 +73,18 @@ If you want to clone the repository and work with the source code:
 
     git co git://github.com/arachni/arachni-rpc-em.git
     cd arachni-rpc-em
-    rake install
+    bundle install
 
 ## Running the Specs
 
-    rake spec
+    bundle exec rake spec
+
+**Warning**: Some of the test cases include stress-testing, don't be alarmed
+when RAM usage hits 5GB and CPU utilization hits 100%.
 
 ## Bug reports/Feature requests
 
-Please send your feedback using Github's issue system at
+Please send your feedback using GitHub's issue system at
 [http://github.com/arachni/arachni-rpc-em/issues](http://github.com/arachni/arachni-rpc-em/issues).
 
 

data/Rakefile

@@ -7,7 +7,10 @@
 =end
 
 require 'rubygems'
-require File.expand_path( File.dirname( __FILE__ ) ) + '/lib/arachni/rpc/em/version'
+require 'bundler'
+require_relative 'lib/arachni/rpc/em/version'
+
+Bundler::GemHelper.install_tasks
 
 begin
     require 'rspec'
@@ -21,39 +24,17 @@ end
 
 task default: [ :build, :spec ]
 
-desc "Generate docs"
+desc 'Generate docs'
 task :docs do
+    outdir = '../arachni-rpc-em-docs'
 
-    outdir = "../arachni-rpc-pages"
     sh "mkdir #{outdir}" if !File.directory?( outdir )
-
-    sh "yardoc --verbose --title \
-      \"Arachni-RPC\" \
-       lib/* -o #{outdir} \
-      - CHANGELOG.md LICENSE.md"
-
-
-    sh "rm -rf .yard*"
-end
-
-desc "Cleaning..."
-task :clean do
-    sh "rm *.gem || true"
+    sh "yardoc -o #{outdir}"
+    sh 'rm -rf .yardoc'
 end
 
-desc "Build the arachni-rpc-em gem."
-task :build => [ :clean ] do
-    sh "gem build arachni-rpc-em.gemspec"
-end
+desc 'Push a new version to RubyGems'
+task :publish => [ :release ]
 
-desc "Build and install the arachni gem."
-task :install  => [ :build ] do
-    sh "gem install arachni-rpc-em-#{Arachni::RPC::EM::VERSION}.gem"
-end
-
-desc "Push a new version to Rubygems"
-task :publish => [ :build ] do
-    sh "git tag -a v#{Arachni::RPC::EM::VERSION} -m 'Version #{Arachni::RPC::EM::VERSION}'"
-    sh "gem push arachni-rpc-em-#{Arachni::RPC::EM::VERSION}.gem"
-end
-task :release => [ :publish ]
+desc 'Build Arachni and run all the tests.'
+task :default => [ :build, :spec ]

data/lib/arachni/rpc/em.rb

@@ -14,7 +14,6 @@ require 'fiber'
 require 'arachni/rpc'
 
 require 'yaml'
-YAML::ENGINE.yamler = 'syck'
 
 dir = File.expand_path( File.dirname( __FILE__ ) )
 require File.join( dir, 'em', 'connection_utilities' )

data/lib/arachni/rpc/em/client.rb

@@ -10,143 +10,36 @@ module Arachni
 module RPC
 module EM
 
+require_relative 'client/handler'
+
 #
 # Simple EventMachine-based RPC client.
 #
 # It's capable of:
-# - performing and handling a few thousands requests per second (depending on
-#   call size, network conditions and the like)
-# - TLS encryption
-# - asynchronous and synchronous requests
-# - handling remote asynchronous calls that require a block
 #
-# @author: Tasos "Zapotek" Laskos <tasos.laskos@gmail.com>
+# * Performing and handling a few thousands requests per second (depending on
+#       call size, network conditions and the like)
+# * TLS encryption
+# * Asynchronous and synchronous requests
+# * Handling remote asynchronous calls that require a block
+#
+# @author Tasos "Zapotek" Laskos <tasos.laskos@gmail.com>
 #
 class Client
     include ::Arachni::RPC::Exceptions
 
-    #
-    # Handles EventMachine's connection and RPC related stuff.
-    #
-    # It's responsible for TLS, storing and calling callbacks as well as
-    # serializing, transmitting and receiving objects.
-    #
-    # @author: Tasos "Zapotek" Laskos <tasos.laskos@gmail.com>
-    #
-    class Handler < EventMachine::Connection
-        include ::Arachni::RPC::EM::Protocol
-        include ::Arachni::RPC::EM::ConnectionUtilities
-
-        DEFAULT_TRIES = 9
-
-        def initialize( opts )
-            @opts = opts.dup
-
-            @max_retries = @opts[:max_retries] || DEFAULT_TRIES
-
-            @opts[:tries] ||= 0
-            @tries ||= @opts[:tries]
+    # Default amount of connections to maintain in the re-use pool.
+    DEFAULT_CONNECTION_POOL_SIZE = 1
 
-            @status = :idle
-
-            @request = nil
-            assume_client_role!
-        end
-
-        def post_init
-            @status = :active
-            start_ssl
-        end
-
-        def unbind( reason )
-            end_ssl
-
-            if @request && @request.callback && !done?
-                if retry? #&& reason == Errno::ECONNREFUSED
-                    retry_request
-                else
-                    e = Arachni::RPC::Exceptions::ConnectionError.new( "Connection closed [#{reason}]" )
-                    @request.callback.call( e )
-                end
-            end
-
-            @status = :closed
-        end
-
-        def connection_completed
-            @status = :established
-        end
-
-        def status
-            @status
-        end
-
-        def done?
-            !!@done
-        end
-
-        #
-        # Used to handle responses.
-        #
-        # @param    [Arachni::RPC::EM::Response]    res
-        #
-        def receive_response( res )
-            if exception?( res )
-                res.obj = Arachni::RPC::Exceptions.from_response( res )
-            end
-
-            @request.callback.call( res.obj ) if @request.callback
-        ensure
-            @done = true
-            @status = :done
-            close_connection
-        end
-
-        def retry_request
-            opts = @opts.dup
-            opts[:tries] += 1
-
-            @tries += 1
-            ::EM.next_tick {
-                ::EM::Timer.new( 0.2 ) {
-                    ::EM.connect( opts[:host], opts[:port], self.class, opts ).
-                        send_request( @request )
-                }
-            }
-        end
-
-        def retry?
-            @tries < @max_retries
-        end
-
-        # @param    [Arachni::RPC::EM::Response]    res
-        def exception?( res )
-            res.obj.is_a?( Hash ) && res.obj['exception'] ? true : false
-        end
-
-        #
-        # Sends the request.
-        #
-        # @param    [Arachni::RPC::EM::Request]      req
-        #
-        def send_request( req )
-            @request = req
-            super( req )
-            @status = :pending
-        end
-    end
-
-    #
-    # Options hash
-    #
-    # @return   [Hash]
-    #
+    # @return   [Hash]  Options hash.
     attr_reader :opts
 
+    attr_reader :connection_count
+
     #
     # Starts EventMachine and connects to the remote server.
     #
-    # opts example:
+    # @example Example options:
     #
     #    {
     #        :host  => 'localhost',
@@ -179,37 +72,143 @@ class Client
     #    }
     #
     # @param    [Hash]  opts
+    # @option   opts    [String]    :host   Hostname/IP address.
+    # @option   opts    [Integer]   :port   Port number.
+    # @option   opts    [String]   :socket   Path to UNIX domain socket.
+    # @option   opts    [Integer]   :connection_pool_size (1)
+    #   Amount of connections to keep open.
+    # @option   opts    [String]    :token  Optional authentication token.
+    # @option   opts    [.dump, .load]      :serializer (YAML)
+    #   Serializer to use for message transmission.
+    # @option   opts    [.dump, .load]      :fallback_serializer
+    #   Optional fallback serializer to be used when the primary one fails.
+    # @option   opts    [Integer]   :max_retries
+    #   How many times to retry failed requests.
+    # @option   opts    [String]    :ssl_ca  SSL CA certificate.
+    # @option   opts    [String]    :ssl_pkey  SSL private key.
+    # @option   opts    [String]    :ssl_cert  SSL certificate.
     #
     def initialize( opts )
         @opts  = opts.merge( role: :client )
         @token = @opts[:token]
 
-        @host, @port = @opts[:host], @opts[:port].to_i
+        @host, @port = @opts[:host], @opts[:port]
+        @socket = @opts[:socket]
+
+        if !@socket && !(@host || @port)
+            fail ArgumentError, 'Needs either a :socket or :host and :port options.'
+        end
+
+        @port = @port.to_i
+
+        if @host && @port <= 0
+            fail ArgumentError, "Invalid port: #{@port}"
+        end
+
+        if @socket && !File.exist?( @socket )
+            fail ArgumentError, "Socket path not valid: #{@socket}"
+        end
+
+        @pool_size = @opts[:connection_pool_size] || DEFAULT_CONNECTION_POOL_SIZE
+
+        @connections      = ::EM::Queue.new
+        @connection_count = 0
 
         Arachni::RPC::EM.ensure_em_running
     end
 
+    # Connection factory, will re-use or create new connections as needed to
+    # accommodate the workload.
+    #
+    # @param    [Block] block   Block to be passed a {Handler connection}.
+    #
+    # @return   [Boolean]
+    #   `true` if a new connection had to be established, `false` if an existing
+    #   one was re-used.
+    def connect( &block )
+        if @connections.empty? && @connection_count < @pool_size
+            #p 'NEW'
+            #p connection_count
+            begin
+                opts = @socket ? @socket : [@host, @port]
+                block.call ::EM.connect( *[opts, Handler, @opts.merge( client: self )].flatten )
+                increment_connection_counter
+            rescue => e
+                block.call e
+            end
+            return true
+        end
+
+        pop_block = proc do |conn|
+            # Some connections may have died while they were waiting in the
+            # queue, get rid of them and start all over in case the queue has
+            # been emptied.
+            if !conn.done?
+                #p 'NOT DONE'
+                #p connection_count
+                connection_failed conn
+                connect( &block )
+                next
+            end
+
+            block.call conn
+        end
+
+        #p 'REUSE'
+        #p connection_count
+        #p @connections.size
+        @connections.pop( &pop_block )
+
+        false
+    end
+
+    def increment_connection_counter
+        @connection_count += 1
+    end
+
+    # {Handler#done? Finished} {Handler}s push themselves here to be re-used.
+    #
+    # @param    [Handler]   connection
+    def push_connection( connection )
+        #p 'PUSHING BACK'
+        #p connection_count
+        #p @connections.size
+        #return if @pool_size <= 0 || @connections.size > @pool_size
+        @connections << connection
+    end
+
+    # Handles failed connections.
+    #
+    # @param    [Handler]   connection
+    def connection_failed( connection )
+        #p 'CON FAILED'
+        #p connection_count
+        #p @connections.size
+        @connection_count -= 1
+        connection.close_without_retry
+    end
+
     #
     # Calls a remote method and grabs the result.
     #
     # There are 2 ways to perform a call, async (non-blocking) and sync (blocking).
     #
-    # To perform an async call you need to provide a block which will be passed
-    # the return value once the method has finished executing.
+    # @example To perform an async call you need to provide a block to handle the result.
     #
     #    server.call( 'handler.method', arg1, arg2 ) do |res|
     #        do_stuff( res )
     #    end
     #
     #
-    # To perform a sync (blocking) call do not pass a block, the value will be
-    # returned as usual.
+    # @example To perform a sync (blocking), call without a block.
     #
     #    res = server.call( 'handler.method', arg1, arg2 )
     #
-    # @param    [String]    msg     in the form of <i>handler.method</i>
-    # @param    [Array]     args    collection of arguments to be passed to the method
-    # @param    [Proc]      &block
+    # @param    [String]    msg
+    #   RPC message in the form of `handler.method`.
+    # @param    [Array]     args
+    #   Collection of arguments to be passed to the method.
+    # @param    [Block]      block
     #
     def call( msg, *args, &block )
         req = Request.new(
@@ -224,22 +223,26 @@ class Client
 
     private
 
-    def connect
-        ::EM.connect( @host, @port, Handler, @opts )
+    def set_exception( req, e )
+        exc = ConnectionError.new( e.to_s + " for '#{@host}:#{@port}'." )
+        exc.set_backtrace e.backtrace
+        req.callback.call exc
     end
 
     def call_async( req, &block )
-        ::EM.next_tick {
-            req.callback = block if block_given?
+        req.callback = block if block_given?
 
-            begin
-                connect.send_request( req )
-            rescue ::EM::ConnectionError => e
-                exc = ConnectionError.new( e.to_s + " for '#{@host}:#{@port}'." )
-                exc.set_backtrace( e.backtrace )
-                req.callback.call exc
+        begin
+            connect do |connection|
+                if connection.is_a? Exception
+                    set_exception( req, connection )
+                    next
+                end
+                connection.send_request( req )
             end
-        }
+        rescue => e
+            set_exception( req, e )
+        end
     end
 
     def call_sync( req )
@@ -253,6 +256,7 @@ class Client
                 t.wakeup
                 ret = obj
             end
+            raise ret if ret.is_a?( Exception )
             sleep
         else
             f = Fiber.current
@@ -262,11 +266,11 @@ class Client
                 ret = Fiber.yield
             rescue FiberError => e
                 msg = e.to_s + "\n"
-                msg += '(Consider wrapping your sync code in a' +
-                    ' "::Arachni::RPC::EM::Synchrony.run" ' +
-                    'block when your app is running inside the Reactor\'s thread)'
+                msg += '(Consider wrapping your sync code in a' <<
+                    ' "::Arachni::RPC::EM::Synchrony.run" block when your app ' <<
+                    'is running inside the Reactor\'s thread)'
 
-                raise( msg )
+                raise msg
             end
         end