ffi-rzmq 0.8.2 → 0.9.0

data/AUTHORS.txt

@@ -16,3 +16,4 @@ Stefan Kaes, github: skaes
 
 Dmitry Ustalov, github: eveel
 
+Patrik Sundberg, github: sundbp

data/History.txt

@@ -1,3 +1,38 @@
+== 0.9.0 / 20110930
+  * Changed the behavior of every method that used to produce exceptions.
+    The methods now behave more like the C API functions. They return
+    result codes instead of raising exceptions. Further, the "receive"
+    methods on Socket now all take an empty string as a buffer to read
+    the message into.
+    This is a BREAKING CHANGE and is NOT backward compatible with earlier
+    releases. I apologize for the inconvenience, but this API will be
+    much easier to test/spec and maintain. It will also allow for the
+    production of more logical code.
+    
+  * Major refactoring of Socket internals so that a single gem can
+    support libzmq 2.x, 3.x and 4.x APIs without any user intervention.
+    The correct libzmq version is detected at runtime and used to
+    configure the Ruby classes to conform to the proper API.
+    
+  * Added Socket#recvmsgs as a convenience method for receiving a
+    multipart message into an array of Messages.
+    
+  * Added support for new 0mq API introduced in the 3.0 branch. 
+    API mostly changed for sending and receiving messages with new
+    POSIX-compliant send() and recv() functions. The original
+    functions were renamed sendmsg() and recvmsg().
+    Additionally, most getsockopt() and setsockopt() calls now use
+    an int (4 bytes) instead of a mish-mash of 32-bit and 64-bit
+    values.
+    For a full list of differences, visit the 0mq wiki page at:
+    http://www.zeromq.org/docs:3-0-upgrade
+    
+  * Created a new ext/ directory so that users can copy the libzmq*
+    library files directly into the gem for easier distribution. This
+    path is checked *before* the usual system paths.
+    
+  * Rewrote all examples to use the revised API.
+
 == 0.8.2 / 20110728
   * Fixed major bug with Socket#setsockopt when writing 8-byte longs.
 

data/README.rdoc

@@ -9,12 +9,44 @@ function interface). It's a pure ruby wrapper so this gem can be loaded
 and run by any ruby runtime that supports FFI. That's all of them:
 MRI 1.9.x, Rubinius and JRuby.
 
+This single gem supports 0mq 2.x and 3.x 0mq APIs. The 0mq project started
+making backward-incompatible changes to the API with the 3.x release.
+The gem auto-configures itself to expose the API conforming to the loaded
+C library.
+
 The impetus behind this library was to provide support for ZeroMQ in
-JRuby which has native threads. Unlike MRI, IronRuby and
-Rubinius which all have a GIL, JRuby allows for threaded access to ruby
-code from outside extensions. ZeroMQ is heavily threaded, so until the
-other runtimes remove their GIL, JRuby will likely be the best
-environment to run this library.
+JRuby which has native threads. Unlike MRI, which has a GIL, JRuby and
+Rubinius allow for threaded access to Ruby code from outside extensions. 
+ZeroMQ is heavily threaded, so until the MRI runtime removes its GIL, 
+JRuby and Rubinius will likely be the best environments to run this library.
+
+== Breaking API Changes from 0.8.x to 0.9.0
+
+There has been a major change in API from the 0.8.x series to 0.9.0. 
+Previously the Socket#send and Socket#recv methods could raise exceptions
+when there was an error. They also returned true or false depending on
+the success of failure of the operation. Mixing and matching return codes
+and exceptions is a terrible idea in practice, so all of the exception
+code from Socket has been removed.
+
+The Poller and Message class have also had their exceptions removed for
+returning errors on instance methods.
+
+Unfortunately, it isn't possible to indicate that a call to #new has failed
+by returning nil. A call to #new that fails needs to raise an exception to
+indicate failure. (This isn't strictly true; this behavior can be changed
+but it is not a standard Ruby idiom and is therefore a surprising, and
+poor, thing to do.) Classes that formerly raised exceptions during allocation
+now offer a factory method #create that will either return a successfully
+constructed object or nil. Code should check for nil returns before using
+the object.
+
+This is a *breaking* change. I'm sorry but the old API's inconsistency was
+causing too many problems. It now more closely mimics the 0mq C API. If you
+prefer the original API, it should be relatively simple to wrap these
+classes up to provide the original API.
+
+All example code has been updated to use the new Ruby API.
 
 == PERFORMANCE
 
@@ -24,19 +56,22 @@ Check out the latest performance results:
 
 == FEATURES/PROBLEMS:
 
-This gem needs more tests. I'm certain there are a
-ton of bugs, so please open issues for them here or fork this project,
-fix them, and send me a pull request.
+This gem needs more tests. This gem has been battle tested by myself
+and others for over a year, so I am fairly confident that it is solid.
+However, it is inevitable that there will be bugs, so please open 
+issues for them here or fork this project, fix them, and send me a pull
+request.
 
 The 'ffi' gem has dropped support for MRI 1.8.x. Since this project relies
 on that gem to load and run this code, then this project also no longer
-supports MRI 1.8.x. 
+supports MRI 1.8.x. I recommend JRuby for the best performance and
+stability.
 
 All features are implemented.
 
 == SYNOPSIS:
 
-Client code:
+0mq API v2 client code:
   
   require 'rubygems'
   require 'ffi-rzmq'
@@ -61,10 +96,8 @@ Client code:
     s.send_string msg, 0
   end
   
-  # give the lib time to flush any remaining messages
-  sleep 1
 
-Server code:
+0mq API v2 server code:
 
   require 'rubygems'
   require 'ffi-rzmq'
@@ -102,7 +135,7 @@ I highly recommend visiting the Learn Ruby 0mq project for a bunch of good code
 
 == REQUIREMENTS:
 
-  * 0mq 2.0.10 or 2.1+
+  * 0mq 2.1.x or later; 2.0.x is no longer supported
   
 The ZeroMQ library must be installed on your system in a well-known location
 like /usr/local/lib. This is the default for new ZeroMQ installs.
@@ -140,7 +173,7 @@ To build from git master:
 
 (The MIT License)
 
-Copyright (c) 2010 Chuck Remes
+Copyright (c) 2011 Chuck Remes
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/Rakefile

@@ -13,11 +13,16 @@ namespace :win do
     rm_rf PKG_PATH
     system "gem build #{NAME}.gemspec"
     mkdir_p PKG_PATH
-    mv "#{NAME}-0.7.1.gem", PKG_PATH
-    system "gem install #{PKG_PATH}/#{NAME}-0.7.1.gem"
+    mv "#{NAME}-*.gem", PKG_PATH
+    system "gem install #{PKG_PATH}/#{NAME}-*.gem"
   end
 end
 
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec
+
 Bones {
   name 'ffi-rzmq'
   authors 'Chuck Remes'

data/examples/README.rdoc

@@ -2,17 +2,21 @@
 
 == Requirements
 
-1. Installed gem
+1. lib dir
 
-All of the examples assume the gem has been successfully installed.
+All of the examples assume the lib directory containing the gem sources is two directories up from the location of the example.
 
 2. Installed libzmq library
 
 The ZeroMQ C libraries need to be downloaded, compiled and installed separately from the gem. Please see http://www.zeromq.org/area:download for links to the downloadable files along with some simple installation instructions. Also, be sure to check the FAQ if you run into problems with compiling.
 
-3. Two terminal windows
+This gem auto-configures itself to conform to the API for 0mq 2.1.x, 3.x and 4.x. The 0mq project started making backward-incompatible changes with the 3.x branch. Rather than create separate gems, this one handles all of them.
 
-ZeroMQ is used to build network applications. At minimum, there is a "client" application and a "server" application that talk to each other over the network, IPC or an internal thread queue. For the sake of code simplicity, these programs are in separate files and need to be executed in different windows.
+It is possible to install the libzmq* files directly into the gem in the ext/ directory. This directory is checked for loadable libraries first before it falls back to checking the system paths.
+
+3. One terminal window
+
+ZeroMQ is used to build network applications. At minimum, there is a "client" application and a "server" application that talk to each other over the network, IPC or an internal thread queue. Several of the examples start the client and server components within separate threads or use the polling mechanism to interleave I/O operations amongst several sockets. A few examples need two terminal windows because the client and server code is in separate files (local_lat.rb/remote_lat.rb, local_throughput.rb/remote_throughput.rb).
 
 == Latency Test
 
@@ -20,34 +24,26 @@ The examples include a latency performance test. The example sets up a pair of R
 
 ==== Files
 
-* local_lat.rb
-* remote_lat.rb
+* latency_measurement.rb
 
 ==== Arguments
 
 The remote_lat.rb program takes 3 arguments:
 
-  [bind_to]  Requires a transport string of the format "transport"://"endpoint"<:><port>. For example, tcp://127.0.0.1:5555
+  [link_address]  Requires a transport string of the format "transport"://"endpoint"<:><port>. For example, tcp://127.0.0.1:5555
   
   [message size] Size of each message measured in bytes. Allowable range is 1 to 2^(64-1).
   
   [message count] The number of round-trips used for the latency measurements. Allowable range is 1 to 2^(64-1).
 
-
-The local_lat.rb program also takes 3 arguments. They should exactly mirror the arguments given to remote_lat.rb.
-
   
 ==== Execution
 
-In one of the terminals, start up the remote_lat.rb program first. It *must* be launched first so that it can be ready and waiting for the first message sent by the local_lat.rb program.
-
-  % ruby remote_lat.rb tcp://127.0.0.1:5555 1024 100_000
+In an open terminal window, execute the latency_measurement.rb file.
 
-In the second terminal window, start up the local_lat.rb program too.
+  % ruby latency_measurement.rb tcp://127.0.0.1:5555 1024 100_000
 
-  % ruby local_lat.rb tcp://127.0.0.1:5555 1024 100_000
-  
-On a relatively new system, it can run 100k messages in under 30 seconds. When complete, the remote_lat.rb program prints out a few statistics and exits. The local_lat.rb program does not print any message when it exits.
+On a relatively new system, it can run 100k messages in under 30 seconds. When complete, the program prints out a few statistics and exits.
 
 Running with a larger "message count" will yield a more accurate latency measurement since nearly all Ruby runtimes require a little warm up time to hit their stride. I recommend 100k as a minimum while 10 million is better for determining a true measure.
 
@@ -55,87 +51,36 @@ On a desktop computer purchased in 2007, all of the Ruby runtimes report a laten
 
 
 
-== Zero Copy Latency Test
-
-The examples include a zero copy latency performance test. The example sets up a pair of REQ/REP sockets and send a message back and forth as fast as possible. There is only a single message in flight at any given moment. The time required to send the message the requested number of times determines overall single-message latency for this type of socket.
-
-The difference between this test and the first latency test has to do with the management of the sent and received messages. This test does not examine or copy the contents of the message at all. Instead, its mere presence is sufficient to echo it back to the other program. Therefore, this test is doing a lot less work for each message send & receive than the first latency test. Also, this test more perfectly mirrors the work being performed by the C latency test.
-
-==== Files
-
-* local_lat_zerocopy.rb
-* remote_lat_zerocopy.rb
-
-==== Arguments
-
-The remote_lat_zerocopy.rb program takes 3 arguments:
-
-  [bind_to]  Requires a transport string of the format "transport"://"endpoint"<:><port>. For example, tcp://127.0.0.1:5555
-  
-  [message size] Size of each message measured in bytes. Allowable range is 1 to 2^(64-1).
-  
-  [message count] The number of round-trips used for the latency measurements. Allowable range is 1 to 2^(64-1).
-
-
-The local_lat_zerocopy.rb program also takes 3 arguments. They should exactly mirror the arguments given to remote_lat_zerocopy.rb.
-
-  
-==== Execution
-
-In one of the terminals, start up the remote_lat_zerocopy.rb program first. It *must* be launched first so that it can be ready and waiting for the first message sent by the local_lat_zerocopy.rb program.
-
-  % ruby remote_lat_zerocopy.rb tcp://127.0.0.1:5555 1024 100_000
-
-In the second terminal window, start up the local_lat.rb program too.
-
-  % ruby local_lat_zerocopy.rb tcp://127.0.0.1:5555 1024 100_000
-  
-On a relatively new system, it can run 100k messages in under 30 seconds. When complete, the remote_lat_zerocopy.rb program prints out a few statistics and exits. The local_lat_zerocopy.rb program does not print any message when it exits.
-
-Running with a larger "message count" will yield a more accurate latency measurement since nearly all Ruby runtimes require a little warm up time to hit their stride. I recommend 100k as a minimum while 10 million is better for determining a true measure.
-
-On a desktop computer purchased in 2007, all of the Ruby runtimes report a latency of approximately 95 microseconds per message. For comparison, the pure C latency test reports approximately 88 microseconds of latency.
-
-
-
 == Throughput Test
 
 The examples include a throughput performance test. The example sets up a pair of PUB/SUB sockets and publish messages as fast as possible to a subscriber listening for every message. The publisher can send much faster than the subscriber can retrieve messages. 
 
-Since the publisher completes first, that program contains a "sleep" statement to keep the program alive and active which gives the subscriber more time to consume the queued messages. When the publisher exits, the socket closes and discards all remaining messages.
+Since the publisher completes first, the program waits for all subscribers to exit before closing the PUB socket. This is necessary because all enqueued messages are discarded when the socket is closed.
 
 The subscriber prints some statistics when it exits.
 
 ==== Files
 
-* local_throughput.rb
-* remote_throughput.rb
+* throughput_measurement.rb
 
 ==== Arguments
 
-The local_throughput.rb program takes 3 arguments:
+The throughput_measurement.rb program takes 3 arguments:
 
-  [bind_to]  Requires a transport string of the format "transport"://"endpoint"<:><port>. For example, tcp://127.0.0.1:5555
+  [link_address]  Requires a transport string of the format "transport"://"endpoint"<:><port>. For example, tcp://127.0.0.1:5555
   
   [message size] Size of each message measured in bytes. Allowable range is 1 to 2^(64-1).
   
   [message count] The number of round-trips used for the latency measurements. Allowable range is 1 to 2^(64-1).
 
 
-The remote_throughput.rb program also takes 3 arguments. They should exactly mirror the arguments given to local_throughput.rb.
-
-  
 ==== Execution
 
-In one of the terminals, start up the local_throughput.rb program first. It *must* be launched first so that it can be ready and waiting for the first message published by the remote_throughput.rb program.
+In an open terminal, execute the throughput_measurement.rb script.
 
-  % ruby local_throughput.rb tcp://127.0.0.1:5555 1024 100_000
+  % ruby throughput_measurement.rb tcp://127.0.0.1:5555 1024 100_000
 
-In the second terminal window, start up the second program.
-
-  % ruby remote_throughput.rb tcp://127.0.0.1:5555 1024 100_000
-  
-On a relatively new system, it can run 100k messages in under 10 seconds. When complete, the local_throughput.rb program prints out a few statistics and exits. The remote_throughput.rb program does not print any message when it exits.
+On a relatively new system, it can run 100k messages in under 10 seconds. When complete, the program prints out a few statistics and exits.
 
 Running with a larger "message count" will yield a more accurate latency measurement since nearly all Ruby runtimes require a little warm up time to hit their stride. I recommend 100k as a minimum while 1 million is better for determining a true measure. NOTE! The publisher can send much faster than the subscriber so the publisher's queue will grow very rapidly in RAM. For 1 million messages (or more) this can consume hundreds of megabytes or gigabytes of RAM. On my system, sending 10 million messages requires 10 GB of RAM before the subscriber can catch up.
 
@@ -145,7 +90,7 @@ On a desktop computer purchased in 2007, all of the Ruby runtimes report a throu
 
 == Poll
 
-For a reasonable example of using zmq_poll(), take a look at the reqrep_poll.rb program. It illustrates the use of zmq_poll(), as wrapped by the Ruby library, for detecting and responding to read and write events recorded on sockets. It also shows how to use ZMQ::NO_BLOCK for non-blocking send and receive.
+For a reasonable example of using zmq_poll(), take a look at the reqrep_poll.rb program. It illustrates the use of zmq_poll(), as wrapped by the Ruby library, for detecting and responding to read and write events recorded on sockets. It also shows how to use ZMQ::NO_BLOCK/ZMQ::DONTWAIT for non-blocking send and receive.
 
 ==== Files
 

data/examples/{local_lat.rb → v2api/local_lat.rb}

@@ -16,11 +16,10 @@
 #    You should have received a copy of the Lesser GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-require 'rubygems'
-require 'ffi-rzmq'
+require File.join(File.dirname(__FILE__), '..', '..', 'lib', 'ffi-rzmq')
 
 if ARGV.length < 3
-  puts "usage: local_lat <connect-to> <message-size> <roundtrip-count>"
+  puts "usage: ruby local_lat.rb <connect-to> <message-size> <roundtrip-count>"
   exit
 end
 
@@ -28,16 +27,32 @@ bind_to = ARGV[0]
 message_size = ARGV[1].to_i
 roundtrip_count = ARGV[2].to_i
 
-ctx = ZMQ::Context.new 1
-s = ctx.socket ZMQ::REP
-s.setsockopt(ZMQ::HWM, 100)
-s.bind(bind_to)
+def assert(rc)
+  raise "Last API call failed at #{caller(1)}" unless rc >= 0
+end
+
+begin
+  ctx = ZMQ::Context.new
+  s = ctx.socket(ZMQ::REP)
+rescue ContextError => e
+  STDERR.puts "Failed to allocate context or socket!"
+  raise
+end
+
+assert(s.setsockopt(ZMQ::LINGER, 100))
+assert(s.setsockopt(ZMQ::HWM, 100))
+
+assert(s.bind(bind_to))
 
 roundtrip_count.times do
-  msg = s.recv_string 0
-  raise "Message size doesn't match, expected [#{message_size}] but received [#{msg.size}]" if message_size != msg.size
-  s.send_string msg, 0
+  string = ''
+  assert(s.recv_string(string, 0))
+
+  raise "Message size doesn't match, expected [#{message_size}] but received [#{string.size}]" if message_size != string.size
+
+  assert(s.send_string(string, 0))
 end
 
-# give the lib time to flush any remaining messages
-sleep 1
+assert(s.close)
+
+ctx.terminate

data/examples/v2api/local_lat_poll.rb

@@ -0,0 +1,66 @@
+
+require File.join(File.dirname(__FILE__), '..', '..', 'lib', 'ffi-rzmq')
+
+if ARGV.length < 3
+  puts "usage: ruby local_lat.rb <connect-to> <message-size> <roundtrip-count>"
+  exit
+end
+
+link = ARGV[0]
+message_size = ARGV[1].to_i
+roundtrip_count = ARGV[2].to_i
+
+def assert(rc)
+  raise "Last API call failed at #{caller(1)}" unless rc >= 0
+end
+
+begin
+  ctx = ZMQ::Context.new
+  s1 = ctx.socket(ZMQ::REQ)
+  s2 = ctx.socket(ZMQ::REP)
+rescue ContextError => e
+  STDERR.puts "Failed to allocate context or socket!"
+  raise
+end
+
+assert(s1.setsockopt(ZMQ::LINGER, 100))
+assert(s2.setsockopt(ZMQ::LINGER, 100))
+
+assert(s1.connect(link))
+assert(s2.bind(link))
+
+poller = ZMQ::Poller.new
+poller.register_readable(s2)
+poller.register_readable(s1)
+
+
+start_time = Time.now
+
+# kick it off
+message = ZMQ::Message.new("a" * message_size)
+assert(s1.send(message, ZMQ::NOBLOCK))
+
+i = roundtrip_count
+
+until i.zero?
+  i -= 1
+
+  assert(poller.poll_nonblock)
+
+  poller.readables.each do |socket|
+    received_message = ''
+    assert(socket.recv_string(received_message, ZMQ::NOBLOCK))
+    assert(socket.send(ZMQ::Message.new(received_message), ZMQ::NOBLOCK))
+  end
+end
+
+elapsed_usecs = (Time.now.to_f - start_time.to_f) * 1_000_000
+latency = elapsed_usecs / roundtrip_count / 2
+
+puts "mean latency: %.3f [us]" % latency
+puts "received all messages in %.3f seconds" % (elapsed_usecs / 1_000_000)
+
+assert(s1.close)
+assert(s2.close)
+
+ctx.terminate