rbczmq 0.1

data/.gitignore

@@ -0,0 +1,23 @@
+.libs/*
+.rbx/*
+*.rbc
+*.lo
+*.la
+*.lai
+*.dylib.dSYM
+*.dylib
+*.o
+*.a
+*.log
+tmp/*
+true/*
+*.bundle
+*.gem
+doc/*
+.DS_Store
+ext/zeromq
+ext/czmq
+pkg
+scratch
+ext/rbczmq/dst
+ext/rbczmq/Makefile

data/.travis.yml

@@ -0,0 +1,19 @@
+language: ruby
+rvm:
+  - rbx-18mode
+  - rbx-19mode
+  - ree
+  - 1.8.7
+  - 1.9.2
+  - 1.9.3
+  - ruby-head
+script: "bundle exec rake"
+before_install: "sudo apt-get install uuid-dev"
+gemfile:
+  - Gemfile
+notifications:
+  recipients:
+    - lourens@methodmissing.com
+branches:
+  only:
+    - master

data/Gemfile

@@ -0,0 +1,5 @@
+source :rubygems
+
+gemspec
+
+gem 'rake'

data/Gemfile.lock

@@ -0,0 +1,19 @@
+PATH
+  remote: .
+  specs:
+    rbczmq (0.1)
+
+GEM
+  remote: http://rubygems.org/
+  specs:
+    rake (0.9.2.2)
+    rake-compiler (0.8.0)
+      rake
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  rake
+  rake-compiler (~> 0.8.0)
+  rbczmq!

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2011 Lourens Naudé - http://github.com/methodmissing
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,247 @@
+= rbczmq - binding for the high level ZeroMQ C API {<img src="https://secure.travis-ci.org/methodmissing/rbczmq.png" alt="Build Status" />}[http://travis-ci.org/methodmissing/rbczmq]
+
+(c) 2011 Lourens Naudé (methodmissing), James Tucker (raggi) with API guidance from the czmq (http://czmq.zeromq.org/) project.
+
+  http://github.com/methodmissing/rbczmq
+
+==== NOT YET PRODUCTION READY, BUT ALMOST
+
+== About ZeroMQ
+
+In a nutshell, ZeroMQ is a hybrid networking library / concurrency framework. I quote the ØMQ Guide (http://zguide.zeromq.org/page:all) :
+
+"ØMQ (ZeroMQ, 0MQ, zmq) looks like an embeddable networking library but acts like a concurrency framework. It gives you sockets that carry whole messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fanout, pub-sub, task distribution, and request-reply. It's fast enough to be the fabric for clustered products. Its asynchronous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks. It has a score of language APIs and runs on most operating systems. ØMQ is from iMatix and is LGPL open source."
+
+== Another ZeroMQ extension ?
+
+This extension bundles both ZeroMQ (libzmq, http://www.zeromq.org/) and CZMQ (libczmq, http://czmq.zeromq.org/) and as such have no third party dependencies other than a Ruby distribution and a C compiler. My goals for this project were :
+
+* Access to a powerful messaging technology without having to install a bunch of dependencies
+* A stable and mostly version agnostic (2.x and 3.x series) API
+* Leverage and build upon a mature and maintained client (CZMQ)
+* Target Ruby distributions with a stable and comprehensive C API (MRI, Rubinius, JRuby is work in progress)
+* Support for running sockets in Threads - both green and native threads should be supported and preempt properly
+  with edge-triggered multiplexing from libzmq.
+* Integrate with the Garbage Collector in a predictable way. CZMQ and the ZeroMQ framework is very fast and can allocate
+  an enormous amount of objects in no time when using the Frame, Message and String wrappers. Resources such as socket
+  connections should be cleaned up when objects are finalized as well.
+* Expose Message envelopes and Frames to developers as well to allow for higher level protocols and constructs.
+* Enforce well known best practices such as restricting socket interactions to within the thread the socket was created in etc.
+
+== Performance
+
+ZeroMQ can have higher throughput than TCP in most cases by using a message batching technique. Please have a look through the Performance section in the ZeroMQ FAQ (http://www.zeromq.org/area:faq#toc2) for further implementation details.
+
+Some notes about these benchmarks :
+
+* Messages go through the full network stack on localhost (TCP/IP transport)
+* The sender and receiver endpoints are Ruby processes which coerce transferred data to native String objects
+  on both ends.
+* There's thus a definite method dispatch cost in addition to intermittent pauses from the Garbage Collector
+* It's still plenty fast for most soft real-time applications and we're able to push in excess of 1 gigabits/s with the 1024 byte payloads. A language with automatic memory management cannot easily comply to hard real-time guarantees anyways.
+
+=== TCP/IP loopback, 100k messages, 100 byte payloads
+
+    Lourenss-MacBook-Air:rbczmq lourens$ MSG_COUNT=100000 MSG_SIZE=100 ruby perf/pair.rb
+    Local pids: 2042
+    Remote pid: 2043
+    Sent 100000 messages in 0.3933s ...
+    [2043] Memory used before: 1128kb
+    [2043] Memory used after: 3012kb
+    [2042] Memory used before: 1120kb
+    ====== [2042] transfer stats ======
+    message encoding: string
+    message size: 100 [B]
+    message count: 100000
+    mean throughput: 227978 [msg/s]
+    mean throughput: 182.383 [Mb/s]
+    [2042] Memory used after: 22432kb
+
+=== TCP/IP loopback, 100k messages, 1024 byte payloads
+
+    Lourenss-MacBook-Air:rbczmq lourens$ MSG_COUNT=100000 MSG_SIZE=1024 ruby perf/pair.rb
+    Local pids: 2027
+    Remote pid: 2028
+    Sent 100000 messages in 0.641198s ...
+    [2028] Memory used before: 1120kb
+    [2028] Memory used after: 12776kb
+    [2027] Memory used before: 1144kb
+    ====== [2027] transfer stats ======
+    message encoding: string
+    message size: 1024 [B]
+    message count: 100000
+    mean throughput: 160756 [msg/s]
+    mean throughput: 1316.919 [Mb/s]
+    [2027] Memory used after: 189004kb
+
+=== TCP/IP loopback, 100k messages, 2048 byte payloads
+
+    Lourenss-MacBook-Air:rbczmq lourens$ MSG_COUNT=100000 MSG_SIZE=2048 ruby perf/pair.rb
+    Local pids: 2034
+    Remote pid: 2035
+    Sent 100000 messages in 0.94703s ...
+    [2035] Memory used before: 1140kb
+    [2035] Memory used after: 7212kb
+    [2034] Memory used before: 1128kb
+    ====== [2034] transfer stats ======
+    message encoding: string
+    message size: 2048 [B]
+    message count: 100000
+    mean throughput: 123506 [msg/s]
+    mean throughput: 2023.528 [Mb/s]
+    [2034] Memory used after: 277712kb
+
+Have a play around with the performance runner and other socket pairs as well - https://github.com/methodmissing/rbczmq/tree/master/perf
+
+== Usage
+
+As a first step I'd highly recommend you read (and reread) through the zguide (http://zguide.zeromq.org/page:all) as understanding the supported messaging patterns and topologies is fundamental to getting the most from this binding. 
+Here's a few basic examples. Please refer to documentation (http://methodmissing.github.com/rbczmq/) and test cases (https://github.com/methodmissing/rbczmq/tree/master/test) for detailed usage information.
+
+=== Basic send / receive, in process transport
+
+    ctx = ZMQ::Context.new
+    rep = ctx.socket(:PAIR)
+    port = rep.bind("inproc://send.receive")
+    req = ctx.socket(:PAIR)
+    req.connect("inproc://send.receive")
+    req.send("ping") # true
+    rep.recv # "ping"
+
+    ctx.destroy
+
+=== Fair-queued work distribution to a set of worker threads
+
+    ctx = ZMQ::Context.new
+    push = ctx.bind(:PUSH, "inproc://push-pull-distribution.test")
+    threads = []
+    5.times do
+      threads << Thread.new do
+        pull = ctx.connect(:PULL, "inproc://push-pull-distribution.test")
+        msg = pull.recv
+        pull.close
+        msg
+      end
+    end
+
+    sleep 0.5 # avoid "slow joiner" syndrome
+    messages = %w(a b c d e f)
+    messages.each do |m|
+      push.send m
+    end
+
+    threads.each{|t| t.join }
+    threads.all?{|t| messages.include?(t.value) } # true
+
+    ctx.destroy
+
+=== Async request / reply routing
+
+    ctx = ZMQ::Context.new
+    router = ctx.bind(:ROUTER, "inproc://routing-flow.test")
+    dealer = ctx.socket(:DEALER)
+    dealer.identity = "xyz"
+    dealer.connect("inproc://routing-flow.test")
+
+    router.sendm("xyz")
+    router.send("request")
+    dealer.recv # "request"
+
+    dealer.send("reply")
+    router.recv # "xyz"
+    router.recv # "reply"
+
+    ctx.destroy
+
+=== Send / receive frames
+
+    ctx = ZMQ::Context.new
+    rep = ctx.socket(:PAIR)
+    rep.bind("inproc://frames.test")
+    req = ctx.socket(:PAIR)
+    req.connect("inproc://frames.test")
+    ping = ZMQ::Frame("ping")
+    req.send_frame(ping) # true
+    rep.recv_frame # ZMQ::Frame("ping")
+    rep.send_frame(ZMQ::Frame("pong")) # true
+    req.recv_frame # ZMQ::Frame("pong")
+    rep.send_frame(ZMQ::Frame("pong")) # true
+    req.recv_frame_nonblock # nil
+    sleep 0.3
+    req.recv_frame_nonblock # ZMQ::Frame("pong")
+
+    ctx.destroy
+
+=== Send / receive messages
+
+    ctx = ZMQ::Context.new
+    rep = ctx.socket(:PAIR)
+    rep.bind("inproc://messages.test")
+    req = ctx.socket(:PAIR)
+    req.connect("inproc://messages.test")
+
+    msg = ZMQ::Message.new
+    msg.push ZMQ::Frame("header")
+    msg.push ZMQ::Frame("body")
+
+    req.send_message(msg) # nil
+
+    recvd_msg = rep.recv_message
+    recvd_msg.class # ZMQ::Message
+    recvd_msg.pop # ZMQ::Frame("header")
+    recvd_msg.pop # ZMQ::Frame("body")
+
+    ctx.destroy
+
+== Resources
+
+* ZeroMQ - http://www.zeromq.org/community
+* The ØMQ Reference Manual - http://api.zeromq.org/
+* The ØMQ FAQ - http://www.zeromq.org/area:faq
+* Whitepapers - http://www.zeromq.org/area:whitepapers
+* The ØMQ Guide - http://zguide.zeromq.org/page:all
+* CZMQ - http://czmq.zeromq.org/
+* Recent presentation on data transfer by the author of this binding - http://www.slideshare.net/methodmissing/sapo-codebits-2011
+
+== Requirements
+
+* A POSIX compliant OS, known to work well on Linux, BSD variants and Mac OS X
+* Ruby MRI 1.8, 1.9 or Rubinius (JRuby capi support forthcoming)
+* A C compiler
+
+== Installation
+
+Rubygems installation
+
+    gem install rbczmq
+
+Building from source
+
+    git clone git@github.com:methodmissing/rbczmq.git
+    rake
+
+Running tests
+
+    rake test
+
+== TODO
+
+* ZMQ::Message#save && ZMQ::Message.load
+* Optimize zloop handler callbacks (perftools)
+* OS X leaks utility - http://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man1/leaks.1.html
+* Handle GC issue with timers in loop callbacks
+* czmq send methods aren't non-blocking by default
+* Enforce socket timeouts
+* Revisit the ZMQ::Loop API
+* Push gem out to rubygems.org
+* RDOC fail on mixed C and Ruby source files that document that same constants
+* GC guards to prevent recycling objects being sent / received.
+* Sockets can bind && connect to multiple endpoints - account for that
+* Watch out for further cases where REQ / REP pairs could raise EFSM
+* Do not clobber local scope from macros (James's commit in master)
+* Support installation without vendor'ed libs as well
+* Incorporate examples into CI as well
+
+== Contact, feedback and bugs
+
+This project is still work in progress and I'm looking for guidance on API design, use cases and any outlier experiences. Please log bugs and suggestions at https://github.com/methodmissing/rbczmq/issues

data/Rakefile

@@ -0,0 +1,67 @@
+# encoding: utf-8
+
+require 'rubygems' unless defined?(Gem)
+require 'rake' unless defined?(Rake)
+
+# Prefer compiled Rubinius bytecode in .rbx/
+ENV["RBXOPT"] = "-Xrbc.db"
+
+require 'rake/extensiontask'
+require 'rake/testtask'
+begin
+require 'rdoc/task'
+rescue LoadError # fallback to older 1.8.7 rubies
+require 'rake/rdoctask'
+end
+
+gemspec = eval(IO.read('rbczmq.gemspec'))
+
+Gem::PackageTask.new(gemspec) do |pkg|
+end
+
+Rake::ExtensionTask.new('rbczmq', gemspec) do |ext|
+  ext.name = 'rbczmq_ext'
+  ext.ext_dir = 'ext/rbczmq'
+
+  CLEAN.include 'ext/rbczmq/dst'
+  CLEAN.include 'ext/zeromq'
+  CLEAN.include 'ext/czmq'
+  CLEAN.include 'lib/**/rbczmq_ext.*'
+end
+
+Rake::RDocTask.new do |rd|
+  files = FileList["README.rdoc", "lib/**/*.rb", "ext/rbczmq/*.c"]
+  rd.title = "rbczmq - binding for the high level ZeroMQ C API"
+  rd.main = "README.rdoc"
+  rd.rdoc_dir = "doc"
+  rd.options << "--promiscuous"
+  rd.rdoc_files.include(files)
+end
+
+desc 'Run rbczmq tests'
+Rake::TestTask.new(:test) do |t|
+  t.test_files = Dir.glob("test/**/test_*.rb")
+  t.verbose = true
+  t.warning = true
+end
+
+namespace :debug do
+  desc "Run the test suite under gdb"
+  task :gdb do
+    system "gdb --args ruby rake"
+  end
+
+  desc "Run the test suite under valgrind"
+  task :valgrind do
+    valgrind_opts = "--num-callers=5 --error-limit=no --partial-loads-ok=yes --undef-value-errors=no --leak-check=full"
+    system %[valgrind #{valgrind_opts} ruby -w -I"lib" -I"/Users/lourens/.rvm/gems/ruby-1.8.7-p352/gems/rake-0.9.2/lib" "/Users/lourens/.rvm/gems/ruby-1.8.7-p352/gems/rake-0.9.2/lib/rake/rake_test_loader.rb" "test/test_context.rb" "test/test_frame.rb" "test/test_loop.rb" "test/test_message.rb" "test/test_socket.rb" "test/test_threading.rb" "test/test_timer.rb" "test/test_zmq.rb"]
+  end
+end
+
+desc 'Clobber Rubinius .rbc files'
+task :clobber_rbc do
+  sh 'find . -name *.rbc -print0 | xargs -0 rm'
+end
+
+task :test => :compile
+task :default => :test

data/examples/loop.rb

@@ -0,0 +1,109 @@
+# encoding: utf-8
+
+$:.unshift('.')
+$:.unshift(File.expand_path(File.dirname(__FILE__)) + '/../lib')
+require 'zmq'
+require 'pp'
+
+class ConsumerHandler < ZMQ::Handler
+  def initialize(pollable, consumer)
+    super
+    @consumer = consumer
+  end
+
+  def on_readable
+    @consumer.perform recv
+  end
+end
+
+class ProducerHandler < ZMQ::Handler
+  def initialize(pollable, producer)
+    super
+    @producer = producer
+  end
+
+  def on_writable
+    @producer.work
+  end
+end
+
+class Consumer
+  attr_reader :thread
+  def initialize(ctx, endpoint, topic = "")
+    @socket = ctx.socket(:SUB)
+    @socket.subscribe("")
+    @socket.connect(endpoint)
+    # verbose output
+    @socket.verbose = true
+    @socket.subscribe(topic)
+    @jobs, @working = 0, 0.0
+  end
+
+  def start
+    ZL.register_readable(@socket, ConsumerHandler, self)
+    self
+  end
+
+  def stop
+    ZL.remove(@socket)
+    stats
+  end
+
+  def perform(work)
+    # Random hot loop to simulate CPU intensive work
+    start = Time.now
+    work.to_i.times{}
+    @jobs += 1
+    @working += (Time.now - start).to_f
+  end
+
+  private
+  def stats
+    puts "Processed #{@jobs} jobs in %.4f seconds" % @working
+    $stdout.flush
+  end
+end
+
+class Producer
+  def initialize(ctx, endpoint, topic = "")
+    @ctx, @endpoint, @topic, @consumers = ctx, endpoint, topic, []
+    @socket = ctx.socket(:PUB)
+    # verbose output
+    @socket.verbose = true
+    @socket.bind(endpoint)
+    @socket.linger = 1
+    @interrupted = false
+  end
+
+  def spawn_consumers(count = 10)
+    count.times do
+      @consumers << Consumer.new(@ctx, @endpoint, @topic).start
+    end
+  end
+
+  def start
+    ZL.register_writable(@socket, ProducerHandler, self)
+  end
+
+  def stop
+    @consumers.each{|c| c.stop }
+    ZL.remove(@socket)
+    ZL.stop
+    @ctx.destroy
+  end
+
+  def work
+    work = "#{@topic}#{rand(100_000).to_s}"
+    @socket.send(work)
+  end
+end
+
+ZL.run do
+  ctx = ZMQ::Context.new
+  producer = Producer.new(ctx, 'inproc://example.loop')
+  producer.spawn_consumers
+  trap(:INT) do
+    producer.stop
+  end
+  producer.start
+end