cztop 1.1.2 → 1.2.0

data/examples/taxi_system/README.md

@@ -1,276 +0,0 @@
-# Taxi System
-
-Suppose you're running a taxi company. You have a set of taxi drivers
-working for you.  You'd like to connect them to your central server, so they're
-ready to get service requests from customers who'd like to get picked up by a
-taxi from some place X. As soon as a customer sends his service request, the
-central server will send the closest taxi nearby that's available to the
-customer.
-
-Of course you want the communication between the broker and the taxi drivers to
-be secure, meaning you want encryption and authentication.
-
-You also want ping-pong heartbeating, as you want to have confidence you can
-get in touch with your taxi drivers any time you want. And if a service
-request can't be delivered to a particular taxi driver, you wanna know
-immediately.
-
-This solution is implemented using CLIENT/SERVER sockets and the CURVE
-security mechanism.
-
-## Broker
-
-Here's a possible implementation of the broker. What you'll have to provide
-are the environment variables `BROKER_ADDRESS` (the public TCP endpoint),
-`BROKER_CERT` (path to the broker's secret+public keys), and `CLIENT_CERTS`
-(directory to taxi drivers' certificates, public keys only).
-
-After the start, the broker will just start listening for the drivers (CLIENT
-sockets) to connect. After a driver has connected, authenticated, and sent its
-`HELLO` message, the broker answers with a `WELCOME` or `WELCOMEBACK` message,
-depending if the driver was connected before (it might have reconnected and
-been assigned a new routing ID).
-
-The broker will present you with a Pry shell. Right before starting the shell,
-there's a small usage information, but it's not very well visible due to Pry's
-noisy start. It's simple, though. Inside that shell, you can use the method
-`#send_command(driver, command)`. Example:
-
-```
-  pry> send_command("driver1", "foobar")
-```
-
-Depending on whether the driver is connected, it'll send the message or report
-that it cannot do so.
-
-```ruby
-#!/usr/bin/env ruby
-require 'pry'
-require 'pathname'
-require 'cztop'
-
-endpoint = ENV["BROKER_ADDRESS"]
-broker_cert = CZTop::Certificate.load ENV["BROKER_CERT"] # secret+public
-client_certs = ENV["CLIENT_CERTS"] # /path/to/client_certs/
-drivers = Pathname.new(client_certs).children.map(&:basename).map(&:to_s)
-
-authenticator = CZTop::Authenticator.new
-authenticator.verbose!
-authenticator.curve(client_certs)
-
-# create and bind socket
-@socket = CZTop::Socket::SERVER.new
-@socket.CURVE_server!(broker_cert)
-#socket.options.sndtimeo = 0
-@socket.options.heartbeat_ivl     = 100#ms
-@socket.options.heartbeat_timeout = 300#ms
-@socket.bind(endpoint)
-
-puts ">>> Socket bound to #{endpoint.inspect}"
-
-# get and print socket events
-Thread.new do
-  monitor = CZTop::Monitor.new(@socket)
-  monitor.listen("ALL")
-  monitor.start
-  while msg = monitor.next
-    puts ">>> Socket event: #{msg.inspect}"
-  end
-end
-
-# receive messages from drivers
-@driver_map = {} # driver name => routing ID
-Thread.new do
-
-  # CZTop::Loop (zloop) doesn't work with SERVER sockets :(
-  poller = CZTop::Poller.new(@socket)
-  while true
-    puts "waiting for socket to become readable ..."
-    socket = poller.wait
-    puts "socket is readable"
-    msg = socket.receive
-    puts "got message"
-    command, argument = msg[0].split("\t", 2)
-
-    case command
-    when "HELLO"
-      driver = argument
-      puts ">>> Driver #{driver.inspect} has connected."
-      welcome = @driver_map.key?(driver) ? "WELCOMEBACK" : "WELCOME"
-
-      # remember driver's assigned message routing ID
-      @driver_map[driver] = msg.routing_id
-
-      # send WELCOME or WELCOMEBACK
-      rep = CZTop::Message.new(welcome)
-      rep.routing_id = @driver_map[driver]
-      socket << rep
-      puts ">>> Sent #{welcome.inspect} to #{driver.inspect}"
-    end
-  end
-end
-
-def send_command(driver, command)
-  if command.nil? || command.empty?
-    puts "!!! No message given."
-    return
-  end
-  if not @driver_map.key?(driver)
-    puts "!!! Driver #{driver.inspect} has never connected."
-    return
-  end
-  puts ">>> Sending message to #{driver.inspect} ..."
-  msg = CZTop::Message.new(command)
-  msg.routing_id = @driver_map[driver]
-  @socket << msg
-rescue SocketError
-  puts "!!! Driver #{driver.inspect} isn't connected anymore."
-end
-
-##
-# REPL for user to play
-#
-puts <<MSG
-You can now send messages to the drivers yourself.
-The use the method #send_command, like this:
-
-pry> send_command("driver1", "PICKUP\t(8.541694,47.376887)")
-
-This should show something like this in the client.rb terminal:
-
-  03:17:01 driver1.1 | received message: "PICKUP\t(8.541694,47.376887)"
-MSG
-
-binding.pry
-```
-
-## Client
-
-Here you have to provide the environment variables `BROKER_ADDRESS` (ditto),
-`BROKER_CERT` (public key only), `CLIENT_CERT` (taxi driver's certificate
-containing the secret+public keys).
-
-After connecting to the broker and completing the security handshake, the
-client sends a `HELLO` message, after which it immediately expects some answer
-from the broker (see above). After that, it just listens for messages (service
-requests) and prints them into the terminal.
-
-```ruby
-#!/usr/bin/env ruby
-require 'cztop'
-
-endpoint = ENV["BROKER_ADDRESS"]
-broker_cert = CZTop::Certificate.load ENV["BROKER_CERT"] # public only
-client_cert = CZTop::Certificate.load ENV["CLIENT_CERT"]
-
-@socket = CZTop::Socket::CLIENT.new
-@socket.CURVE_client!(client_cert, broker_cert)
-@socket.options.sndtimeo     = 2000#ms
-
-# heartbeating:
-# * send PING every 100ms
-# * close connection after 300ms of no life sign from broker
-# * tell broker to close connection after 500ms of no life sign from client
-@socket.options.heartbeat_ivl     = 100#ms
-@socket.options.heartbeat_timeout = 300#ms
-@socket.options.heartbeat_ttl     = 500#ms
-
-@socket.connect(endpoint)
-puts ">>> connected."
-
-# tell broker who we are
-@socket << "HELLO\t#{client_cert["driver_name"]}"
-puts ">>> sent HELLO."
-welcome = @socket.receive[0]
-puts ">>> got #{welcome}."
-
-poller = CZTop::Poller.new(@socket)
-while true
-  socket = poller.wait
-  message = socket.receive
-  puts ">>> received message: #{message[0].inspect}"
-end
-```
-
-## How to run the example
-
-### Generate broker's and drivers' keys
-
-Here's a simple script that'll create the broker's certificate and the taxi
-drivers' certificates. There are also public key only files so a minimum amount
-of information can be made available on one system, e.g. a taxi driver's system
-must not know the broker's secret key. Also, the broker doesn't necessarily
-need to know the clients' secret keys just to authenticate them.
-
-```ruby
-#!/usr/bin/env ruby
-require 'cztop'
-require 'fileutils'
-FileUtils.cd(File.dirname(__FILE__))
-FileUtils.mkdir "public_keys"
-FileUtils.mkdir "public_keys/drivers"
-FileUtils.mkdir "secret_keys"
-FileUtils.mkdir "secret_keys/drivers"
-#FileUtils.mkdir "certs/drivers"
-
-DRIVERS = %w[ driver1 driver2 driver3 ]
-
-# broker certificate
-cert = CZTop::Certificate.new
-cert.save("secret_keys/broker")
-cert.save_public("public_keys/broker")
-
-# driver certificates
-DRIVERS.each do |driver_name|
-  cert = CZTop::Certificate.new
-  cert["driver_name"] = driver_name
-  cert.save "secret_keys/drivers/#{driver_name}"
-  cert.save_public "public_keys/drivers/#{driver_name}"
-end
-
-```
-Run it as follows:
-
-```
-./generate_keys.rb
-```
-
-### Start broker
-
-Run this:
-
-```
-./start_broker.sh
-```
-
-which will execute the following script:
-
-```sh
-#!/bin/sh -x
-BROKER_ADDRESS=tcp://127.0.0.1:4455 BROKER_CERT=secret_keys/broker CLIENT_CERTS=public_keys/drivers ./broker.rb
-```
-
-### Start driver software instances
-
-Run this in another terminal:
-
-```
-./start_clients.sh
-```
-
-which will execute the following script:
-
-```sh
-#!/bin/sh -x
-export BROKER_ADDRESS=tcp://127.0.0.1:4455
-export BROKER_CERT=public_keys/broker
-CLIENT_CERT=secret_keys/drivers/driver1_secret ./client.rb &
-CLIENT_CERT=secret_keys/drivers/driver2_secret ./client.rb &
-CLIENT_CERT=secret_keys/drivers/driver3_secret ./client.rb &
-jobs
-jobs -p
-jobs -l
-trap 'kill $(jobs -p)' EXIT
-wait
-```
-

data/examples/taxi_system/broker.rb

@@ -1,97 +0,0 @@
-#!/usr/bin/env ruby
-require 'pry'
-require 'pathname'
-require 'cztop'
-
-endpoint = ENV["BROKER_ADDRESS"]
-broker_cert = CZTop::Certificate.load ENV["BROKER_CERT"] # secret+public
-client_certs = ENV["CLIENT_CERTS"] # /path/to/client_certs/
-drivers = Pathname.new(client_certs).children.map(&:basename).map(&:to_s)
-
-authenticator = CZTop::Authenticator.new
-authenticator.verbose!
-authenticator.curve(client_certs)
-
-# create and bind socket
-@socket = CZTop::Socket::SERVER.new
-@socket.CURVE_server!(broker_cert)
-#socket.options.sndtimeo = 0
-@socket.options.heartbeat_ivl     = 100#ms
-@socket.options.heartbeat_timeout = 300#ms
-@socket.bind(endpoint)
-
-puts ">>> Socket bound to #{endpoint.inspect}"
-
-# get and print socket events
-Thread.new do
-  monitor = CZTop::Monitor.new(@socket)
-  monitor.listen("ALL")
-  monitor.start
-  while msg = monitor.next
-    puts ">>> Socket event: #{msg.inspect}"
-  end
-end
-
-# receive messages from drivers
-@driver_map = {} # driver name => routing ID
-Thread.new do
-
-  poller = CZTop::Poller.new(@socket)
-  while true
-    puts "waiting for socket to become readable ..."
-    socket = poller.simple_wait
-    puts "socket is readable"
-    msg = socket.receive
-    puts "got message"
-    command, argument = msg[0].split("\t", 2)
-
-    case command
-    when "HELLO"
-      driver = argument
-      puts ">>> Driver #{driver.inspect} has connected."
-      welcome = @driver_map.key?(driver) ? "WELCOMEBACK" : "WELCOME"
-
-      # remember driver's assigned message routing ID
-      @driver_map[driver] = msg.routing_id
-
-      # send WELCOME or WELCOMEBACK
-      rep = CZTop::Message.new(welcome)
-      rep.routing_id = @driver_map[driver]
-      socket << rep
-      puts ">>> Sent #{welcome.inspect} to #{driver.inspect}"
-    end
-  end
-end
-
-def send_command(driver, command)
-  if command.nil? || command.empty?
-    puts "!!! No message given."
-    return
-  end
-  if not @driver_map.key?(driver)
-    puts "!!! Driver #{driver.inspect} has never connected."
-    return
-  end
-  puts ">>> Sending message to #{driver.inspect} ..."
-  msg = CZTop::Message.new(command)
-  msg.routing_id = @driver_map[driver]
-  @socket << msg
-rescue SocketError
-  puts "!!! Driver #{driver.inspect} isn't connected anymore."
-end
-
-##
-# REPL for user to play
-#
-puts <<MSG
-You can now send messages to the drivers yourself.
-The use the method #send_command, like this:
-
-pry> send_command("driver1", "PICKUP\t(8.541694,47.376887)")
-
-This should show something like this in the client.rb terminal:
-
-  03:17:01 driver1.1 | received message: "PICKUP\t(8.541694,47.376887)"
-MSG
-
-binding.pry

data/examples/taxi_system/client.rb

@@ -1,34 +0,0 @@
-#!/usr/bin/env ruby
-require 'cztop'
-
-endpoint = ENV["BROKER_ADDRESS"]
-broker_cert = CZTop::Certificate.load ENV["BROKER_CERT"] # public only
-client_cert = CZTop::Certificate.load ENV["CLIENT_CERT"]
-
-@socket = CZTop::Socket::CLIENT.new
-@socket.CURVE_client!(client_cert, broker_cert)
-@socket.options.sndtimeo     = 2000#ms
-
-# heartbeating:
-# * send PING every 100ms
-# * close connection after 300ms of no life sign from broker
-# * tell broker to close connection after 500ms of no life sign from client
-@socket.options.heartbeat_ivl     = 100#ms
-@socket.options.heartbeat_timeout = 300#ms
-@socket.options.heartbeat_ttl     = 500#ms
-
-@socket.connect(endpoint)
-puts ">>> connected."
-
-# tell broker who we are
-@socket << "HELLO\t#{client_cert["driver_name"]}"
-puts ">>> sent HELLO."
-welcome = @socket.receive[0]
-puts ">>> got #{welcome}."
-
-poller = CZTop::Poller.new(@socket)
-while true
-  socket = poller.simple_wait
-  message = socket.receive
-  puts ">>> received message: #{message[0].inspect}"
-end

data/examples/taxi_system/generate_keys.rb

@@ -1,24 +0,0 @@
-#!/usr/bin/env ruby
-require 'cztop'
-require 'fileutils'
-FileUtils.cd(File.dirname(__FILE__))
-FileUtils.mkdir "public_keys"
-FileUtils.mkdir "public_keys/drivers"
-FileUtils.mkdir "secret_keys"
-FileUtils.mkdir "secret_keys/drivers"
-#FileUtils.mkdir "certs/drivers"
-
-DRIVERS = %w[ driver1 driver2 driver3 ]
-
-# broker certificate
-cert = CZTop::Certificate.new
-cert.save("secret_keys/broker")
-cert.save_public("public_keys/broker")
-
-# driver certificates
-DRIVERS.each do |driver_name|
-  cert = CZTop::Certificate.new
-  cert["driver_name"] = driver_name
-  cert.save "secret_keys/drivers/#{driver_name}"
-  cert.save_public "public_keys/drivers/#{driver_name}"
-end

data/examples/taxi_system/start_broker.sh

@@ -1,2 +0,0 @@
-#!/bin/sh -x
-BROKER_ADDRESS=tcp://127.0.0.1:4455 BROKER_CERT=secret_keys/broker CLIENT_CERTS=public_keys/drivers ./broker.rb

data/examples/taxi_system/start_clients.sh

@@ -1,11 +0,0 @@
-#!/bin/sh -x
-export BROKER_ADDRESS=tcp://127.0.0.1:4455
-export BROKER_CERT=public_keys/broker
-CLIENT_CERT=secret_keys/drivers/driver1_secret ./client.rb &
-CLIENT_CERT=secret_keys/drivers/driver2_secret ./client.rb &
-CLIENT_CERT=secret_keys/drivers/driver3_secret ./client.rb &
-jobs
-jobs -p
-jobs -l
-trap 'kill $(jobs -p)' EXIT
-wait

data/examples/weather_pub_sub/pub.rb

@@ -1,24 +0,0 @@
-#!/usr/bin/env ruby
-#
-# Weather update server, based on ZMQ's zguide.
-# Binds PUB socket to ipc:///tmp/weather_pubsub_example
-# Publishes random weather updates
-#
-
-require 'cztop'
-
-# create and bind socket
-socket = CZTop::Socket::PUB.new("ipc:///tmp/weather_pubsub_example")
-puts "<<< Socket bound to #{socket.last_endpoint.inspect}"
-
-while true
-	# Generate values for zipcodes
-	zipcode = rand(100000)
-	temperature = rand(215) - 80
-	relhumidity = rand(50) + 10
-	
-	update = "%05d %d %d" % [zipcode, temperature, relhumidity]
-	puts update
-	
-	socket << update
-end

data/examples/weather_pub_sub/sub.rb

@@ -1,33 +0,0 @@
-#!/usr/bin/env ruby
-#
-# Weather update client, based on that from ZMQ's zguide.
-# Connects SUB socket to ipc:///tmp/weather_pubsub_example
-# Collects weather updates and finds avg temp in zipcode
-#
-
-require 'cztop'
-
-COUNT = 100
-
-# Create socket, connect to publisher.
-socket = CZTop::Socket::SUB.new("ipc:///tmp/weather_pubsub_example")
-puts ">>> Socket Connected"
-
-# Subscribe to zipcode.  Default: Chicago - 60606
-filter = ARGV.size > 0 ? ARGV[0] : "60606"
-socket.subscribe(filter)
-
-# gather & process COUNT updates.
-print "Gathering #{COUNT} samples."
-total_temp = 0
-1.upto(COUNT) do |update_nbr|
-	msg = socket.receive
-	
-	zipcode, temperature, relhumidity = msg[0].split.map(&:to_i)
-	total_temp += temperature
-	# just to show that we're doing something...
-	print "." if update_nbr % 5 == 0
-end
-print "\n"
-
-puts "Average temperatuer for zipcode #{filter} was #{total_temp / COUNT}F."

data/perf/README.md

@@ -1,80 +0,0 @@
-# Performance Measurement
-
-This directory contains simple performance measurement utilities:
-
-- `inproc_lat.rb` measures the latency of the inproc transport
-- `inproc_thru.rb` measures the throughput of the inproc transport
-- `local_lat.rb` and `remote_lat.rb` measure the latency of other transports
-- `local_thru.rb` and `remote_thru.rb` measure the throughput of other transports (TODO)
-
-## Example Output
-
-On my laptop, it currently looks something like this:
-
-### Latency
-
-over inproc, using 10k roundtrips of a repeatedly allocated 1kb message:
-```
-$ bundle exec ./inproc_lat_reqrep.rb 1_000 10_000
-message size: 1000 [B]
-roundtrip count: 10000
-elapsed time: 0.469 [s]
-average latency: 23.439 [us]<Paste>
-```
-
-over IPC, using 10k roundtrips of a repeatedly allocated 1kb message:
-```
-$ bundle exec ./local_lat.rb ipc:///tmp/cztop-perf 1000 1000 & ./remote_lat.rb ipc:///tmp/cztop-perf 1000 1000
-[3] 58043
-message size: 1000 [B]
-roundtrip count: 1000
-elapsed time: 0.091 [s]
-average latency: 45.482 [us]
-[3]    58043 done       ./local_lat.rb ipc:///tmp/cztop-perf 1000 1000
-```
-
-over local TCP/IP stack, using 10k roundtrips of a repeatedly allocated
-1kb message:
-```
-$ bundle exec ./local_lat.rb tcp://127.0.0.1:55667 1000 1000 & ./remote_lat.rb tcp://127.0.0.1:55667 1000 1000
-[3] 58064
-message size: 1000 [B]
-roundtrip count: 1000
-elapsed time: 0.123 [s]
-average latency: 61.434 [us]
-[3]    58064 done       ./local_lat.rb tcp://127.0.0.1:55667 1000 1000
-```
-
-### Throughput
-
-over inproc, with message sizes from 100 bytes to 100kb, 10,000 each:
-
-```
-$ bundle exec ./inproc_thru.rb 100 10_000
-message size: 100 [B]
-message count: 10000
-elapsed time: 0.270 [s]
-mean throughput: 37093 [msg/s]
-mean throughput: 29.674 [Mb/s]
-
-$ bundle exec ./inproc_thru.rb 1_000 10_000
-message size: 1000 [B]
-message count: 10000
-elapsed time: 0.260 [s]
-mean throughput: 38498 [msg/s]
-mean throughput: 307.987 [Mb/s]
-
-$ bundle exec ./inproc_thru.rb 10_000 10_000
-message size: 10000 [B]
-message count: 10000
-elapsed time: 0.317 [s]
-mean throughput: 31501 [msg/s]
-mean throughput: 2520.102 [Mb/s]
-
-$ bundle exec ./inproc_thru.rb 100_000 10_000
-message size: 100000 [B]
-message count: 10000
-elapsed time: 0.906 [s]
-mean throughput: 11034 [msg/s]
-mean throughput: 8827.440 [Mb/s]
-```

data/perf/inproc_lat.rb

@@ -1,49 +0,0 @@
-#! /usr/bin/env ruby
-require "cztop"
-require "benchmark"
-#require "ruby-prof"
-
-if ARGV.size != 2
-  abort <<MSG
-Usage: #{$0} <message-size> <roundtrip-count>
-MSG
-end
-
-MSG_SIZE = Integer(ARGV[0]) # bytes
-ROUNDTRIP_COUNT = Integer(ARGV[1]) # round trips
-MSG = "X" * MSG_SIZE
-
-Thread.new do
-  s = CZTop::Socket::PAIR.new("@inproc://perf")
-  s.signal
-  ROUNDTRIP_COUNT.times do
-    msg = s.receive
-    raise "wrong message size" if msg.content_size != MSG_SIZE
-    s << msg
-  end
-end
-
-s = CZTop::Socket::PAIR.new(">inproc://perf")
-s.wait
-
-#RubyProf.start
-tms = Benchmark.measure do
-  ROUNDTRIP_COUNT.times do
-    s << MSG
-    msg = s.receive
-    raise "wrong message size" if msg.content_size != MSG_SIZE
-  end
-end
-#rubyprof_result = RubyProf.stop
-
-elapsed = tms.real
-latency = elapsed / (ROUNDTRIP_COUNT * 2) * 1_000_000
-puts "message size: #{MSG_SIZE} [B]"
-puts "roundtrip count: #{ROUNDTRIP_COUNT}"
-puts "elapsed time: %.3f [s]" % elapsed
-puts "average latency: %.3f [us]" % latency
-
-# print a flat profile to text
-#printer = RubyProf::FlatPrinter.new(rubyprof_result)
-#printer.print(STDOUT)
-

data/perf/inproc_thru.rb

@@ -1,42 +0,0 @@
-#! /usr/bin/env ruby
-require "cztop"
-require "benchmark"
-
-if ARGV.size != 2
-  abort <<MSG
-Usage: #{$0} <message-size> <message-count>
-MSG
-end
-
-MSG_SIZE = Integer(ARGV[0]) # bytes
-MSG_COUNT = Integer(ARGV[1]) # number of messages
-MSG = "X" * MSG_SIZE
-
-Thread.new do
-  s = CZTop::Socket::PAIR.new("@inproc://perf")
-  s.signal
-  MSG_COUNT.times do
-    msg = s.receive
-    raise "wrong message size" if msg.content_size != MSG_SIZE
-  end
-end
-
-s = CZTop::Socket::PAIR.new(">inproc://perf")
-s.wait
-
-tms = Benchmark.measure do
-  MSG_COUNT.times do
-    s << MSG
-  end
-end
-
-elapsed = tms.real
-
-throughput = MSG_COUNT / elapsed
-megabits = (throughput * MSG_SIZE * 8) / 1_000_000
-
-puts "message size: #{MSG_SIZE} [B]"
-puts "message count: #{MSG_COUNT}"
-puts "elapsed time: %.3f [s]" % elapsed
-puts "mean throughput: %d [msg/s]" % throughput
-puts "mean throughput: %.3f [Mb/s]" % megabits