pipeline_toolkit 1.0.4 → 1.0.6

data/LICENSE.markdown

@@ -0,0 +1,20 @@
+Copyright (c) 2009 Visfleet Ltd.
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.markdown

@@ -0,0 +1,98 @@
+# Pipeline Toolkit #
+by VisFleet
+
+Command line tools for processing messages by constructing a pipeline of workers. [AMQP](http://amqp.rubyforge.org/ "AMQP") and Unix pipes are used to construct the pipeline. Messages are simple Hashes (serialized) so they can hold any values and change throughout the processing.
+
+Provides:
+
+  * Processing acknowledgments, ensuring the a message is only disposed of once it has been successful processed
+  * Performance. Messages are moved through the pipeline fast
+  * Command line tools for:
+    * Subscribing to messages from an [AMQP queue](http://amqp.rubyforge.org/classes/MQ/Queue.html "MQ::Queue")
+    * Pushing messages back onto an [AMQP exchange](http://amqp.rubyforge.org/classes/MQ/Exchange.html "MQ::Exchange") 
+    * Monitoring performance (see msg_probe)
+  * A base module ({MessageCommand}) to include into your own classes to quickly make workers.
+
+## Install ##
+
+  > (sudo) gem install pipeline_toolkit
+
+### Dependancies ###
+
+It is assumed that you have:
+- An [AMQP](http://amqp.rubyforge.org/ "AMQP") message server to pop and push messages to (e.g. http://www.rabbitmq.com/)
+- A *nix system, such as Linux or Mac OS X.
+
+## Usage ##
+
+1. Create your worker
+
+        class MyMachine
+          include MessageCommand
+          
+          def initialize_machine
+            # Optional: setup any dependencies required by your machine
+          end
+          
+          def report_back
+            # Optional: write information about the pipeline back
+            #           through the acknowledgement pipe
+          end
+          
+          def process_standard(message)
+            # Required: handle the message within your machine
+            unless failed_to_process?
+              pass_on(message)  # or alternatively acknowledge if this
+                                # is the last machine in the pipeline.
+            else
+              # handle the failed message here (however you deem fit)
+              acknowledge(message)
+            end
+          end
+        end
+        
+        MyMachine.new.start
+        
+2. Hook it up to a [AMQP](http://amqp.rubyforge.org/ "AMQP") message source
+
+  > msg_subscribe -x raw -t topic -q octane | my_worker.rb | msg_push -x standard -q standard
+
+You can learn more about the command line tools and what options are available by using their help command.
+  
+  > msg_subscriber --help  
+  > msg_push --help  
+  > msg_sink --help  
+  > msg_probe --help  
+
+## Examples ##
+
+In the example, open two terminal windows and then execute the following commands in each:
+  
+  > msg_generator | msg_push -x test-exchange  
+  > msg_subscribe -a false --http-port 9090 -x test-exchange -q test_queue > /dev/null
+  
+The example does the following:
+
+  1. Generate a bunch of sample messages 
+  2. Push them to an [AMQP exchange](http://amqp.rubyforge.org/classes/MQ/Exchange.html "MQ::Exchange")   
+     'test-exchange'. 
+  3. Hook up a pipe to consume the generated messages 
+  4. Push results to /dev/null. 
+  5. Goto http://localhost:9090 to see message throughput.
+
+Now lets switch on acknowledgements:
+
+  > msg_generator | msg_push -x test-exchange  
+  > msg_subscribe -x test-exchange -q test_queue | msg_sink
+
+This example does the same as above, but this time with acknowledgements switched on. 
+This guarantees that a message isn't removed from the message server until handled. 
+msg_sink acknowledges all messages.
+
+## Todo ##
+
+  * DNS-SD is broken. See http://github.com/tenderlove/dnssd/issues#issue/3. We need to fix it or kill it.
+
+  * Write units tests.
+  
+  * Tidy up logging. Not used consistently throughout code. What do we want to log? 

data/bin/msg_generator

@@ -0,0 +1,13 @@
+#!/usr/bin/env ruby
+#
+#  Created on 2010-3-17.
+#  Copyright Visfleet Ltd. (c) 2010. All rights reserved.
+
+require 'rubygems'
+require File.dirname(__FILE__) + '/../lib/pipeline_toolkit'
+require "pipeline_toolkit/commands/msg_generator/cli"
+
+# print an options summary if no args specified
+ARGV << "--help"   if ARGV.empty?
+
+Commands::MsgGenerate::CLI.execute(STDOUT, ARGV)

data/bin/msg_probe

@@ -0,0 +1,13 @@
+#!/usr/bin/env ruby
+#
+#  Created on 2010-3-17.
+#  Copyright Visfleet Ltd. (c) 2010. All rights reserved.
+
+require 'rubygems'
+require File.dirname(__FILE__) + '/../lib/pipeline_toolkit'
+require 'pipeline_toolkit/commands/msg_probe/cli'
+
+# print an options summary if no args specified
+ARGV << "--help"   if ARGV.empty?
+
+Commands::MsgProbe::CLI.execute(STDOUT, ARGV)

data/bin/msg_push

@@ -0,0 +1,13 @@
+#!/usr/bin/env ruby
+#
+#  Created on 2010-3-17.
+#  Copyright Visfleet Ltd. (c) 2010. All rights reserved.
+
+require 'rubygems'
+require File.dirname(__FILE__) + '/../lib/pipeline_toolkit'
+require "pipeline_toolkit/commands/msg_push/cli"
+
+# print an options summary if no args specified
+ARGV << "--help"   if ARGV.empty?
+
+Commands::MsgPush::CLI.execute(STDOUT, ARGV)

data/bin/msg_sink

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+#
+#  Created on 2010-3-17.
+#  Copyright Visfleet Ltd. (c) 2010. All rights reserved.
+
+require 'rubygems'
+require File.dirname(__FILE__) + '/../lib/pipeline_toolkit'
+require "pipeline_toolkit/commands/msg_sink/cli"
+
+# print an options summary if no args specified
+ARGV << "--help"   if ARGV.empty?
+
+Commands::MsgSink::CLI.execute(STDOUT, ARGV)
+

data/bin/msg_subscribe

@@ -0,0 +1,13 @@
+#!/usr/bin/env ruby
+#
+#  Created on 2010-3-17.
+#  Copyright Visfleet Ltd. (c) 2010. All rights reserved.
+
+require 'rubygems'
+require File.dirname(__FILE__) + '/../lib/pipeline_toolkit'
+require "pipeline_toolkit/commands/msg_subscribe/cli"
+
+# print an options summary if no args specified
+ARGV << "--help"   if ARGV.empty?
+
+Commands::MsgSubscribe::CLI.execute(STDOUT, ARGV)

data/lib/pipeline_toolkit.rb

@@ -1,3 +1,13 @@
+$:.unshift(File.dirname(__FILE__)) unless $:.include?(File.dirname(__FILE__)) || $:.include?(File.expand_path(File.dirname(__FILE__)))
+
+require 'pipeline_toolkit/amqp/abstract'
+require 'pipeline_toolkit/amqp/reader'
+require 'pipeline_toolkit/amqp/writer'
+
+require 'pipeline_toolkit/monitoring/monitor_server'
+
+require 'pipeline_toolkit/handlers/message_handler'
+
 require "pipeline_toolkit/default_logger"
 require "pipeline_toolkit/message_coder"
 require "pipeline_toolkit/message_command"
@@ -7,3 +17,10 @@ require "pipeline_toolkit/message_subscriber"
 require "pipeline_toolkit/message_sink"
 require "pipeline_toolkit/open_hash"
 require "pipeline_toolkit/socket_util"
+require 'pipeline_toolkit/message_generator'
+
+require 'pipeline_toolkit/commands/msg_subscribe/cli'
+require 'pipeline_toolkit/commands/msg_generator/cli'
+require 'pipeline_toolkit/commands/msg_probe/cli'
+require 'pipeline_toolkit/commands/msg_push/cli'
+require 'pipeline_toolkit/commands/msg_sink/cli'

data/lib/pipeline_toolkit/amqp/abstract.rb

@@ -0,0 +1,95 @@
+require 'mq'
+
+module Amqp # :nodoc:
+  
+  ##
+  # Provides abstract base functionality used in both {Amqp::Reader} and {Amqp::Writer}
+  #
+  module Abstract
+    
+    ##
+    # Getter for the options hash
+    #
+    attr_reader :options
+
+    ##
+    # Initialize a new instance of either the {Amqp::Reader} or {Amqp::Writer}
+    #
+    # @param options<Hash> The options hash
+    # @option options [Symbol] :host ('localhost') The AMQP server address for the AMQP server.
+    # @option options [Symbol] :port (5672) The AMQP server port for the AMQP server.
+    # @option options [Symbol] :user ('guest') The username as defined by the AMQP server. 
+    # @option options [Symbol] :pass ('guest') The password for the associated :user as defined by the AMQP server.  
+    # @option options [Symbol] :vhost ('/') The virtual host as defined by the AMQP server.  
+    # @option options [Symbol] :type ('fanout') The exchange type (direct, fanout or topic). 
+    # @option options [Symbol] :exchange ('') The exchange name
+    # @option options [Symbol] :durable (false) If set to true, the exchange will be marked as durable.
+    # @option options [Symbol] :passive (false)  If set to true, the server will not create the exchange if it does not already exist.
+    # @option options [Symbol] :ack (true)  If this field is set to false the server does not expect acknowledgments for messages.
+    # @option options [Symbol] :env ('development') The environment to run (development, production).  
+    # 
+    def initialize(options = {})
+      super(options)
+
+      DefaultLogger.init_logger(options)
+
+      @options = options
+    end
+
+    ##
+    # Create a new connection to the AMQP server.
+    #
+    def initialize_connection
+      DefaultLogger.debug("Amqp::Abstract#initialize_connection") if options[:env] == "development"
+      @connection = AMQP.connect(options.select_keys(:host, :port, :user, :pass, :vhost))
+    end
+    
+    ##
+    # Returns a new channel. A channel is a bidirectional virtual connection between the client 
+    # and the AMQP server.
+    #
+    def initialize_channel
+      DefaultLogger.debug("Amqp::Abstract#initialize_channel") if options[:env] == "development"
+      @channel = MQ.new(@connection)
+    end
+
+    ##
+    # Defines, intializes and returns an Exchange 
+    # to act as an ingress point for all published messages. 
+    #
+    def initialize_exchange
+      DefaultLogger.debug("Amqp::Abstract#initialize_exchange") if options[:env] == "development"
+      # declare a exchange on the channel
+      @exchange = MQ::Exchange.new(@channel, options[:type], options[:exchange], :durable => options[:durable], :passive => options[:passive])
+    rescue MQ::Error => e # rescued here because main thread does not seem to see it
+      DefaultLogger.error "#{e.class.name}: #{e.message}\n" << e.backtrace.join("\n")
+      raise e
+    end
+    
+    ##
+    # Defines, intializes and returns an Queue that store and forward messages.
+    #
+    # @param name<String> The name to use for the queue
+    # 
+    def initialize_queue(name = '')
+      DefaultLogger.debug("Amqp::Abstract#initialize_queue(name = '')") if options[:env] == "development"
+      name = options[:queue] if name.nil? || name.empty?
+      @queue = MQ::Queue.new(@channel, name, :durable => options[:durable], :passive => options[:passive])
+    end
+
+    ##
+    # Binds the queue to an exchange.
+    #
+    # @param routing_key<Sting> Specifies the routing key for the binding. The routing key is used for 
+    #                           routing messages depending on the exchange configuration.
+    # 
+    def bind_queue(routing_key = '')
+      DefaultLogger.debug("Amqp::Abstract#bind_queue(routing_key = '')") if options[:env] == "development"
+      unless routing_key.nil? || routing_key.empty?
+        @queue.bind(@exchange, :key => routing_key)
+      else
+        @queue.bind(@exchange)
+      end
+    end
+  end
+end

data/lib/pipeline_toolkit/amqp/reader.rb

@@ -0,0 +1,64 @@
+require 'pipeline_toolkit/amqp/abstract'
+
+module Amqp # :nodoc:
+  
+  ##
+  # The Reader provides functionality specific to asynchronous message 
+  # delivery by subscribing to the AMQP server.
+  #
+  module Reader
+    include Amqp::Abstract
+    
+    ##
+    # Initialize the AMQP server specific to handling 
+    # of messages from the server.
+    #
+    def initialize_reader
+      DefaultLogger.debug("Amqp::Reader#initialize_reader") if options[:env] == "development"
+      @ack_headers ||= {}
+      initialize_connection
+      initialize_channel
+      @channel.prefetch(1) # We only want to handle one message at a time
+      initialize_exchange
+      initialize_queue
+    end
+    
+    ##
+    # Subscribe the queue to receive messages
+    #
+    def queue_subscribe
+      DefaultLogger.debug("Amqp::Reader#queue_subscribe") if options[:env] == "development"
+      unless AMQP.closing?
+        @queue.subscribe(:ack => options[:ack]) do |header, body|
+          DefaultLogger.debug("Rabbit Message: #{body}") if options[:env] == "development"
+          process_queue_message(header, body)
+        end
+      end
+    end
+    
+    ##
+    # Store the acknowledgement header 
+    #
+    # @param message<Hash> The message
+    # @param header<MQ::Header> The message queue header for the message
+    # 
+    def store_acknowledgement(message, header)
+      DefaultLogger.debug("Amqp::Reader#store_acknowledgement(message, header)") if options[:env] == "development"
+      message[:ack_id] = header.delivery_tag.to_s
+      @ack_headers[message[:ack_id]] = header
+    end
+    
+    ##
+    # Perform the acknowledgement on the stored header to 
+    # tell AMQP server we are done with the message.
+    #
+    # @param ack_id<String> The acknowledgement identifier
+    # 
+    def perform_acknowledgement(ack_id)
+      DefaultLogger.debug("Amqp::Reader#perform_acknowledgement(ack_id)") if options[:env] == "development"
+      header = @ack_headers.delete(ack_id)
+      header.ack
+    end
+    
+  end
+end

data/lib/pipeline_toolkit/amqp/writer.rb

@@ -0,0 +1,54 @@
+require 'pipeline_toolkit/amqp/abstract'
+require 'pipeline_toolkit/message_coder'
+
+module Amqp # :nodoc:
+  
+  ##
+  # The Writer provides functionality specific to asynchronous message 
+  # publishing to the AMQP server.
+  #
+  module Writer
+    include Amqp::Abstract
+   
+    ##
+    # Initialize the AMQP server specific to handling 
+    # of messages publishing to the server.
+    #
+    def initialize_writer
+      DefaultLogger.debug("Amqp::Writer#initialize_writer") if options[:env] == "development"
+      initialize_connection
+      initialize_channel
+      initialize_exchange
+    end
+    
+    ##
+    # Initializes the queues to publish the messages too.
+    #
+    def initialize_queues
+      DefaultLogger.debug("Amqp::Writer#initialize_queues") if options[:env] == "development"
+      options[:queues].each do |name, routing_key|
+        initialize_queue(name)
+        bind_queue(routing_key)
+      end
+    end
+    
+    ##
+    # Handles the publishing of messages into the AMQP server.
+    #
+    # @param message<Hash> The message to publish
+    # 
+    def publish(message)
+      DefaultLogger.debug("Amqp::Writer#publish(message)") if options[:env] == "development"
+      if message.has_key?(:routing_key)
+        @exchange.publish(MessageCoder.encode(message), :key => message[:routing_key])
+      else
+        @exchange.publish(MessageCoder.encode(message))
+      end
+    end
+    
+    def queue_names
+      options[:queues].map { |name, type| name }
+    end
+
+  end
+end

data/lib/pipeline_toolkit/commands/msg_generator/cli.rb

@@ -0,0 +1,45 @@
+require 'pp'
+require 'trollop'
+require 'eventmachine'
+
+Signal.trap('INT') do
+  # puts "\nStopping"
+  # TODO: complete signal trap interrupt
+end
+
+Signal.trap('TERM') do
+  # puts "\nStopping"
+  # TODO: complete signal trap terminate
+end
+
+module Commands # :nodoc:
+  module MsgGenerate
+    class CLI
+      def self.execute(stdout, arguments=[])
+
+        opts = Trollop::options do
+          banner <<-EOL
+Message Subscriber
+------------------
+  Generates messages for testing the pipeline. Messages are specified 
+  using JSON.
+
+  Usage:
+  msg_generator [options]
+
+Examples:
+  msg_generator -m '{"id":1, "firstName":"John Smith"}'
+
+Options:
+            EOL
+          opt :msg, "The message to send", :short => "m", :default => '{"id":1}'
+          opt :delay, "Sleep time between sends in seconds", :short => "d", :type => :float
+
+          opt :env, "The environment to run (development, production)", :default => "development", :short => :e
+        end
+
+        MessageGenerator.new(opts).start
+      end
+    end
+  end
+end