pipeline_toolkit 1.1.0 → 1.2.0

data/LICENSE.markdown

@@ -1,4 +1,4 @@
-Copyright (c) 2009 Visfleet Ltd.
+Copyright (c) 2009 VisFleet Ltd.
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.markdown

@@ -1,7 +1,7 @@
 # Pipeline Toolkit #
 by VisFleet
 
-Command line tools for processing messages by constructing a pipeline of workers. [AMQP](http://amqp.rubyforge.org/ "AMQP") and Unix pipes are used to construct the pipeline. Messages are simple Hashes (serialized) so they can hold any values and change throughout the processing.
+Command line tools for processing messages by constructing a pipeline of machines. [AMQP](http://amqp.rubyforge.org/ "AMQP") and Unix pipes are used to construct the pipeline. Messages are simple Hashes (serialized as JSON) so they can hold any values and change throughout the processing.
 
 Provides:
 
@@ -11,21 +11,27 @@ Provides:
     * Subscribing to messages from an [AMQP queue](http://amqp.rubyforge.org/classes/MQ/Queue.html "MQ::Queue")
     * Pushing messages back onto an [AMQP exchange](http://amqp.rubyforge.org/classes/MQ/Exchange.html "MQ::Exchange") 
     * Monitoring performance (see msg_probe)
-  * A base module ({MessageCommand}) to include into your own classes to quickly make workers.
+  * A base module ({MessageCommand}) to include into your own classes to quickly make machines.
 
 ## Install ##
 
-  > (sudo) gem install pipeline_toolkit
+     > (sudo) gem install pipeline_toolkit
 
 ### Dependancies ###
 
 It is assumed that you have:
-- An [AMQP](http://amqp.rubyforge.org/ "AMQP") message server to pop and push messages to (e.g. http://www.rabbitmq.com/)
-- A *nix system, such as Linux or Mac OS X.
+
+  - An [AMQP](http://amqp.rubyforge.org/ "AMQP") message server to pop and push messages to (e.g. http://www.rabbitmq.com/)
+
+  - A *nix system, such as Linux or Mac OS X.
+
+  - The 'uuidgen' command-line tool. This is installed by default on OS X, and is available through most package managers (e.g it is the "uuid-runtime" package in aptitude).   
+
+All gem dependancies are installed automatically, but if you're curious checkout *pipeline_toolkit.gemspec*
 
 ## Usage ##
 
-1. Create your worker
+1. Create your machine
 
         class MyMachine
           include MessageCommand
@@ -34,11 +40,6 @@ It is assumed that you have:
             # Optional: setup any dependencies required by your machine
           end
           
-          def report_back
-            # Optional: write information about the pipeline back
-            #           through the acknowledgement pipe
-          end
-          
           def process_standard(message)
             # Required: handle the message within your machine
             unless failed_to_process?
@@ -55,7 +56,7 @@ It is assumed that you have:
         
 2. Hook it up to a [AMQP](http://amqp.rubyforge.org/ "AMQP") message source
 
-  > msg_subscribe -x raw -t topic -q octane | my_worker.rb | msg_push -x standard -q standard
+  > msg_subscribe -x raw -t topic -q octane | my_machine.rb | msg_push -x standard -q standard
 
 You can learn more about the command line tools and what options are available by using their help command.
   
@@ -89,13 +90,31 @@ This example does the same as above, but this time with acknowledgements switche
 This guarantees that a message isn't removed from the message server until handled. 
 msg_sink acknowledges all messages.
 
-## Todo ##
+## Pre-canned cucumber steps ##
 
-  * DNS-SD is broken. See http://github.com/tenderlove/dnssd/issues#issue/3. We need to fix it or kill it.
+Pipeline toolkit also includes pre-canned cucumber steps for you to test your machines with. The following steps are included:
 
-  * Write units tests.
-    - Make tests re-usable
-  
-  * Tidy up logging. Not used consistently throughout code. What do we want to log? 
+        Given I run this machine:
+        When I input these messages:
+        Then these messages are passed on:
+        Then these messages are acknowledged:
+    
+        Given an amqp server running at "localhost" on port 1234
+        Given amqp queue "test" is empty
+        When I input these messages to exchange "test"
+        Then these messages are on queue "test"
+
+The steps are defined in the following files, so have a look at them to get more details 
+   
+    lib/pipeline_toolkit/cucumber/machine_steps.rb
+    lib/pipeline_toolkit/cucumber/amqp_steps.rb
+
+To make the steps available to your application's cucumber features simply put the follow at the top of your env.rb file (or any file under features/support).
+
+    require 'pipeline_toolkit/cucumber'
 
-  * Change encoding to use YAJL
+## Todo ##
+
+  * Tidy up logging. Feels horrible that we're insisting that SysLogger is used. Want to abstract this out and let user decide which logging framework to use. Also we're hard coding that logs are output to ~/pipeline_toolkit. yuk.
+  
+  * Add tmp path as a argument (hardcoded to /tmp at mo)

data/bin/msg_generator

@@ -3,8 +3,7 @@
 #  Created on 2010-3-17.
 #  Copyright Visfleet Ltd. (c) 2010. All rights reserved.
 
-# Magic line to make bin work from 
-$LOAD_PATH.unshift(File.dirname(__FILE__) + '/../lib') unless $LOAD_PATH.include?(File.dirname(__FILE__) + '/../lib')
-require "pipeline_toolkit/commands/msg_generator/cli"
+$LOAD_PATH.unshift(File.expand_path(File.dirname(__FILE__) + '/../lib')) unless $LOAD_PATH.include?(File.expand_path(File.dirname(__FILE__) + '/../lib'))
+require "pipeline_toolkit/cli/msg_generator_cli"
 
-Commands::MsgGenerate::CLI.execute(STDOUT, ARGV)
+PipelineToolkit::CLI::MsgGeneratorCLI.execute(STDOUT, ARGV)

data/bin/msg_push

@@ -3,11 +3,10 @@
 #  Created on 2010-3-17.
 #  Copyright Visfleet Ltd. (c) 2010. All rights reserved.
 
-# require 'rubygems'
-require File.dirname(__FILE__) + '/../lib/pipeline_toolkit'
-require "pipeline_toolkit/commands/msg_push/cli"
+$LOAD_PATH.unshift(File.expand_path(File.dirname(__FILE__) + '/../lib')) unless $LOAD_PATH.include?(File.expand_path(File.dirname(__FILE__) + '/../lib'))
+require "pipeline_toolkit/cli/msg_push_cli"
 
 # print an options summary if no args specified
 ARGV << "--help"   if ARGV.empty?
 
-Commands::MsgPush::CLI.execute(STDOUT, ARGV)
+PipelineToolkit::CLI::MsgPushCLI.execute(STDOUT, ARGV)

data/bin/msg_sink

@@ -3,12 +3,8 @@
 #  Created on 2010-3-17.
 #  Copyright Visfleet Ltd. (c) 2010. All rights reserved.
 
-require 'rubygems'
-require File.dirname(__FILE__) + '/../lib/pipeline_toolkit'
-require "pipeline_toolkit/commands/msg_sink/cli"
+$LOAD_PATH.unshift(File.expand_path(File.dirname(__FILE__) + '/../lib')) unless $LOAD_PATH.include?(File.expand_path(File.dirname(__FILE__) + '/../lib'))
+require "pipeline_toolkit/cli/msg_sink_cli"
 
-# print an options summary if no args specified
-ARGV << "--help"   if ARGV.empty?
-
-Commands::MsgSink::CLI.execute(STDOUT, ARGV)
+PipelineToolkit::CLI::MsgSinkCLI.execute(STDOUT, ARGV)
 

data/bin/msg_subscribe

@@ -3,11 +3,10 @@
 #  Created on 2010-3-17.
 #  Copyright Visfleet Ltd. (c) 2010. All rights reserved.
 
-require 'rubygems'
-require File.dirname(__FILE__) + '/../lib/pipeline_toolkit'
-require "pipeline_toolkit/commands/msg_subscribe/cli"
+$LOAD_PATH.unshift(File.expand_path(File.dirname(__FILE__) + '/../lib')) unless $LOAD_PATH.include?(File.expand_path(File.dirname(__FILE__) + '/../lib'))
+require "pipeline_toolkit/cli/msg_subscribe_cli"
 
 # print an options summary if no args specified
 ARGV << "--help"   if ARGV.empty?
 
-Commands::MsgSubscribe::CLI.execute(STDOUT, ARGV)
+PipelineToolkit::CLI::MsgSubscribeCLI.execute(STDOUT, ARGV)

data/lib/pipeline_toolkit.rb

@@ -1,5 +1,3 @@
-$:.unshift(File.dirname(__FILE__)) unless $:.include?(File.dirname(__FILE__)) || $:.include?(File.expand_path(File.dirname(__FILE__)))
-
 require 'pipeline_toolkit/amqp/abstract'
 require 'pipeline_toolkit/amqp/reader'
 require 'pipeline_toolkit/amqp/writer'
@@ -14,6 +12,8 @@ require "pipeline_toolkit/message_command"
 require "pipeline_toolkit/message_pusher"
 require "pipeline_toolkit/message_subscriber"
 require "pipeline_toolkit/message_sink"
-require "pipeline_toolkit/open_hash"
-require "pipeline_toolkit/socket_util"
-require 'pipeline_toolkit/message_generator'
+require 'pipeline_toolkit/message_generator'
+
+require "pipeline_toolkit/util/hash_ext"
+require "pipeline_toolkit/util/indifferent_hash"
+require "pipeline_toolkit/util/socket_util"

data/lib/pipeline_toolkit/amqp/abstract.rb

@@ -1,95 +1,98 @@
 require 'mq'
 
-module Amqp # :nodoc:
+module PipelineToolkit
+  module Amqp # :nodoc:
   
-  ##
-  # Provides abstract base functionality used in both {Amqp::Reader} and {Amqp::Writer}
-  #
-  module Abstract
-    
     ##
-    # Getter for the options hash
+    # Provides abstract base functionality used in both {Amqp::Reader} and {Amqp::Writer}
     #
-    attr_reader :options
+    module Abstract
+    
+      ##
+      # Getter for the options hash
+      #
+      attr_reader :options
 
-    ##
-    # Initialize a new instance of either the {Amqp::Reader} or {Amqp::Writer}
-    #
-    # @param options<Hash> The options hash
-    # @option options [Symbol] :host ('localhost') The AMQP server address for the AMQP server.
-    # @option options [Symbol] :port (5672) The AMQP server port for the AMQP server.
-    # @option options [Symbol] :user ('guest') The username as defined by the AMQP server. 
-    # @option options [Symbol] :pass ('guest') The password for the associated :user as defined by the AMQP server.  
-    # @option options [Symbol] :vhost ('/') The virtual host as defined by the AMQP server.  
-    # @option options [Symbol] :type ('fanout') The exchange type (direct, fanout or topic). 
-    # @option options [Symbol] :exchange ('') The exchange name
-    # @option options [Symbol] :durable (false) If set to true, the exchange will be marked as durable.
-    # @option options [Symbol] :passive (false)  If set to true, the server will not create the exchange if it does not already exist.
-    # @option options [Symbol] :ack (true)  If this field is set to false the server does not expect acknowledgments for messages.
-    # @option options [Symbol] :env ('development') The environment to run (development, production).  
-    # 
-    def initialize(options = {})
-      super(options)
+      ##
+      # Initialize a new instance of either the {Amqp::Reader} or {Amqp::Writer}
+      #
+      # @param options<Hash> The options hash
+      # @option options [Symbol] :host ('localhost') The AMQP server address for the AMQP server.
+      # @option options [Symbol] :port (5672) The AMQP server port for the AMQP server.
+      # @option options [Symbol] :user ('guest') The username as defined by the AMQP server. 
+      # @option options [Symbol] :pass ('guest') The password for the associated :user as defined by the AMQP server.  
+      # @option options [Symbol] :vhost ('/') The virtual host as defined by the AMQP server.  
+      # @option options [Symbol] :type ('fanout') The exchange type (direct, fanout or topic). 
+      # @option options [Symbol] :exchange ('') The exchange name
+      # @option options [Symbol] :durable (false) If set to true, the exchange will be marked as durable.
+      # @option options [Symbol] :passive (false)  If set to true, the server will not create the exchange if it does not already exist.
+      # @option options [Symbol] :ack (true)  If this field is set to false the server does not expect acknowledgments for messages.
+      # @option options [Symbol] :env ('development') The environment to run (development, production).  
+      # 
+      def initialize(options = {})
+        super(options)
 
-      DefaultLogger.init_logger(options)
+        DefaultLogger.init_logger(options)
 
-      @options = options
-    end
+        @options = options
+      end
 
-    ##
-    # Create a new connection to the AMQP server.
-    #
-    def initialize_connection
-      DefaultLogger.debug("Amqp::Abstract#initialize_connection") if options[:env] == "development"
-      @connection = AMQP.connect(options.select_keys(:host, :port, :user, :pass, :vhost))
-    end
+      ##
+      # Create a new connection to the AMQP server.
+      #
+      def initialize_connection
+        DefaultLogger.debug("Amqp::Abstract#initialize_connection") if options[:env] == "development"
+        @connection = AMQP.connect(options.select_keys(:host, :port, :user, :pass, :vhost))
+      end
     
-    ##
-    # Returns a new channel. A channel is a bidirectional virtual connection between the client 
-    # and the AMQP server.
-    #
-    def initialize_channel
-      DefaultLogger.debug("Amqp::Abstract#initialize_channel") if options[:env] == "development"
-      @channel = MQ.new(@connection)
-    end
+      ##
+      # Returns a new channel. A channel is a bidirectional virtual connection between the client 
+      # and the AMQP server.
+      #
+      def initialize_channel
+        DefaultLogger.debug("Amqp::Abstract#initialize_channel") if options[:env] == "development"
+        @channel = MQ.new(@connection)
+      end
 
-    ##
-    # Defines, intializes and returns an Exchange 
-    # to act as an ingress point for all published messages. 
-    #
-    def initialize_exchange
-      DefaultLogger.debug("Amqp::Abstract#initialize_exchange") if options[:env] == "development"
-      # declare a exchange on the channel
-      @exchange = MQ::Exchange.new(@channel, options[:type], options[:exchange], :durable => options[:durable], :passive => options[:passive])
-    rescue MQ::Error => e # rescued here because main thread does not seem to see it
-      DefaultLogger.error "#{e.class.name}: #{e.message}\n" << e.backtrace.join("\n")
-      raise e
-    end
+      ##
+      # Defines, intializes and returns an Exchange 
+      # to act as an ingress point for all published messages. 
+      #
+      def initialize_exchange
+        DefaultLogger.debug("Amqp::Abstract#initialize_exchange") if options[:env] == "development"
+        # declare a exchange on the channel
+        @exchange = MQ::Exchange.new(@channel, options[:type], options[:exchange], :durable => options[:durable], :passive => options[:passive])
+      rescue MQ::Error => e # rescued here because main thread does not seem to see it
+        DefaultLogger.error "#{e.class.name}: #{e.message}\n" << e.backtrace.join("\n")
+        raise e
+      end
     
-    ##
-    # Defines, intializes and returns an Queue that store and forward messages.
-    #
-    # @param name<String> The name to use for the queue
-    # 
-    def initialize_queue(name = '')
-      DefaultLogger.debug("Amqp::Abstract#initialize_queue(name = '')") if options[:env] == "development"
-      name = options[:queue] if name.nil? || name.empty?
-      @queue = MQ::Queue.new(@channel, name, :durable => options[:durable], :passive => options[:passive])
-    end
+      ##
+      # Defines, intializes and returns an Queue that store and forward messages.
+      #
+      # @param name<String> The name to use for the queue
+      # 
+      def initialize_queue(name = '')
+        DefaultLogger.debug("Amqp::Abstract#initialize_queue(name = '')") if options[:env] == "development"
+        name = options[:queue] if name.nil? || name.empty?
+        @queue = MQ::Queue.new(@channel, name, :durable => options[:durable], :passive => options[:passive])
+      end
 
-    ##
-    # Binds the queue to an exchange.
-    #
-    # @param routing_key<Sting> Specifies the routing key for the binding. The routing key is used for 
-    #                           routing messages depending on the exchange configuration.
-    # 
-    def bind_queue(routing_key = '')
-      DefaultLogger.debug("Amqp::Abstract#bind_queue(routing_key = '')") if options[:env] == "development"
-      unless routing_key.nil? || routing_key.empty?
-        @queue.bind(@exchange, :key => routing_key)
-      else
-        @queue.bind(@exchange)
+      ##
+      # Binds the queue to an exchange.
+      #
+      # @param routing_key<Sting> Specifies the routing key for the binding. The routing key is used for 
+      #                           routing messages depending on the exchange configuration.
+      # 
+      def bind_queue(routing_key = '')
+        DefaultLogger.debug("Amqp::Abstract#bind_queue(routing_key = '')") if options[:env] == "development"
+        unless routing_key.nil? || routing_key.empty?
+          @queue.bind(@exchange, :key => routing_key)
+        else
+          @queue.bind(@exchange)
+        end
       end
     end
+
   end
 end

data/lib/pipeline_toolkit/amqp/reader.rb

@@ -1,65 +1,68 @@
 require 'pipeline_toolkit/amqp/abstract'
 
-module Amqp # :nodoc:
+module PipelineToolkit
+  module Amqp # :nodoc:
   
-  ##
-  # The Reader provides functionality specific to asynchronous message 
-  # delivery by subscribing to the AMQP server.
-  #
-  module Reader
-    include Amqp::Abstract
-    
     ##
-    # Initialize the AMQP server specific to handling 
-    # of messages from the server.
+    # The Reader provides functionality specific to asynchronous message 
+    # delivery by subscribing to the AMQP server.
     #
-    def initialize_reader
-      DefaultLogger.debug("Amqp::Reader#initialize_reader") if options[:env] == "development"
-      @ack_headers ||= {}
-      initialize_connection
-      initialize_channel
-      @channel.prefetch(1) # We only want to handle one message at a time
-      initialize_exchange
-      initialize_queue
-      bind_queue
-    end
+    module Reader
+      include Amqp::Abstract
     
-    ##
-    # Subscribe the queue to receive messages
-    #
-    def queue_subscribe
-      DefaultLogger.debug("Amqp::Reader#queue_subscribe") if options[:env] == "development"
-      unless AMQP.closing?
-        @queue.subscribe(:ack => options[:ack]) do |header, body|
-          DefaultLogger.debug("Rabbit Message: #{body}") if options[:env] == "development"
-          process_queue_message(header, body)
+      ##
+      # Initialize the AMQP server specific to handling 
+      # of messages from the server.
+      #
+      def initialize_reader
+        DefaultLogger.debug("Amqp::Reader#initialize_reader") if options[:env] == "development"
+        @ack_headers ||= {}
+        initialize_connection
+        initialize_channel
+        @channel.prefetch(1) # We only want to handle one message at a time
+        initialize_exchange
+        initialize_queue
+        bind_queue
+      end
+    
+      ##
+      # Subscribe the queue to receive messages
+      #
+      def queue_subscribe
+        DefaultLogger.debug("Amqp::Reader#queue_subscribe") if options[:env] == "development"
+        unless AMQP.closing?
+          @queue.subscribe(:ack => options[:ack]) do |header, body|
+            DefaultLogger.debug("Rabbit Message: #{body}") if options[:env] == "development"
+            process_queue_message(header, body)
+          end
         end
       end
-    end
     
-    ##
-    # Store the acknowledgement header 
-    #
-    # @param message<Hash> The message
-    # @param header<MQ::Header> The message queue header for the message
-    # 
-    def store_acknowledgement(message, header)
-      DefaultLogger.debug("Amqp::Reader#store_acknowledgement(message, header)") if options[:env] == "development"
-      message[:ack_id] = header.delivery_tag.to_s
-      @ack_headers[message[:ack_id]] = header
-    end
+      ##
+      # Store the acknowledgement header 
+      #
+      # @param message<Hash> The message
+      # @param header<MQ::Header> The message queue header for the message
+      # 
+      def store_acknowledgement(message, header)
+        DefaultLogger.debug("Amqp::Reader#store_acknowledgement(message, header)") if options[:env] == "development"
+        message[:ack_id] = header.delivery_tag.to_s
+        @ack_headers[message[:ack_id]] = header
+      end
     
-    ##
-    # Perform the acknowledgement on the stored header to 
-    # tell AMQP server we are done with the message.
-    #
-    # @param ack_id<String> The acknowledgement identifier
-    # 
-    def perform_acknowledgement(ack_id)
-      DefaultLogger.debug("Amqp::Reader#perform_acknowledgement(ack_id)") if options[:env] == "development"
-      header = @ack_headers.delete(ack_id)
-      header.ack
-    end
+      ##
+      # Perform the acknowledgement on the stored header to 
+      # tell AMQP server we are done with the message.
+      #
+      # @param ack_id<String> The acknowledgement identifier
+      # 
+      def perform_acknowledgement(ack_id)
+        DefaultLogger.debug("Amqp::Reader#perform_acknowledgement(ack_id)") if options[:env] == "development"
+        header = @ack_headers.delete(ack_id)
+        header.ack
+      end
     
+    end
+
   end
 end