mbus 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+--- 
+SHA1: 
+  metadata.gz: 689df9b9f0b86245507faeda6239d9ca24ff2cf1
+  data.tar.gz: 6e95ce8920e6f92951dc28dd967e7049d16ad160
+SHA512: 
+  metadata.gz: 9d8ada5f134ebf4c992d831ac28ad3cc786c40172c22e1279668afbf774858daeb8f7f9b03dd55ad549cff0ab91b36972c74630e16535f8735ea97b6ed80d19c
+  data.tar.gz: cdfe47c4cfbaa25d92a072cd67776b2aa0ff79f145223176c71764154a4845f3be32d0ff2fed93d4c1f357d2d7b0d4dcbcddb2fb37fc6c7ab39bc2c86c27e4a6

data/README.mediawiki

@@ -0,0 +1,169 @@
+= Messagebus =
+
+A gem for publishing and consuming hornetq messagebus messages.
+
+== Installation ==
+
+  $ gem install mbus
+
+== Dependencies ==
+
+This Gem depends upon the following:
+
+=== Runtime ===
+
+* messagebus (vendored an old version for now)
+* thrift
+* json
+
+=== Development & Tests ===
+
+* bundle install
+* Execute rspec spec
+
+== Configuration ==
+
+The messagebus is configured via a single yaml file.  The configuration fields are defined below and an
+example of a full configuration also provided.
+
+* enable_auto_init_connections: turn on producer connections once initialized
+* log_file: path for log
+* log_level: logger level minimum
+* logger: pass a logger directly to the messagebus client
+* worker_log_file: path for worker log
+* cluster_defaults
+** user: hornetq user
+** passwd: hornetq password
+** receipt_wait_timeout_ms: TODO
+** conn_lifetime_sec: TODO
+* clusters: array of haproxy clusters
+** type: producer-cluster or consumer-cluster
+** address: host and port of haproxy delimited by colon
+** destinations: array of queue and topic destinations
+* swarm_config
+** fork: true or false -- whether to use forks or threads when booting drones
+* workers
+** destination: the destination name
+** worker: the worker class to process messages with
+** subscription_id: swarm_drone_run_spec_id
+** ack_on_error: whether to send an ack for a job that failed
+** drones: number of drones to boot up for parallel processing
+
+'''Example config'''
+  enable_auto_init_connections: true 
+  log_file: log/messagebus-client.log
+  worker_log_file: log/messagebus-consumer.log
+  
+  :swarm_config:
+    # :fork: true
+  
+  :workers:
+    -
+      :destination: jms.topic.testTopic2
+      :worker: SimpleDroneRunSpecWorker
+      :subscription_id: swarm_drone_run_spec_id
+      
+      :ack_on_error: false
+      :drones: 1
+  
+  :cluster_defaults:
+    :user: guest
+    :passwd: guest
+    :receipt_wait_timeout_ms: 1500
+    :conn_lifetime_sec: 300
+    :enable_dynamic_serverlist_fetch: true  
+  
+  clusters:
+    -
+      name: orders-messagebus-cluster
+      producer_address: localhost:61613
+      consumer_address: localhost:61613
+      destinations:
+        - jms.topic.testTopic2
+
+== Getting Started ==
+
+You will find an example listed of loading a configuration, instantiation of a client, and
+finally invocation of the publish method.  There are three different message types that can
+be published on the bus, string, binary, or json.  An example of each are listed with concrete
+code to use.
+
+In its current state, the messagebus gem should be vendored into your application.  This will
+change once the internal rubygems server is integrated into the company.  For now, unpack the
+gem and include "thrift" 0.9.0 into your Gemfile next to the messagebus declaration.
+
+=== Edit Gemfile ===
+
+  gem "thrift", "0.9.0"
+  gem "messagebus", "x.x.x", :path => "vendor/gems/messagebus-x.x.x", :require => "messagebus"
+
+
+=== Bundle your gems ===
+
+  $ bundle
+
+== Producing Messages ==
+=== Run interactive ruby ===
+
+  $ irb
+
+  ruby>
+  require "messagebus" # Notice this will happen by default in the bundler
+
+  ruby>
+  binary_to_publish = "\xfe\x3e\x5e"
+  config = YAML.load_file("./config/messagebus.yml")
+  client = Messagebus::Client.new(config.merge(:logger => Logger.new("a/directory/to/my/file.log")))  
+  client.start
+  client.publish "jms.queue.testQueue1", binary_to_publish
+
+  ruby>
+  string_to_publish = "the quick brown fox jumped over the lazy dogs back"
+  config = YAML.load_file("./config/messagebus.yml")
+  client = Messagebus::Client.new(config.merge(:logger => Logger.new("a/directory/to/my/file.log")))  
+  client.publish "jms.queue.testQueue1", string_to_publish
+
+  ruby>
+  hash_to_publish = { :ruby => { :loves => :the_bus } }
+  config = YAML.load_file("./config/messagebus.yml")
+  client = Messagebus::Client.new(config.merge(:logger => Logger.new("a/directory/to/my/file.log")))  
+  client.publish "jms.queue.testQueue1", hash_to_publish
+
+  ruby>
+  object_to_publish = DomainObjectThatRespondsToToJson # object_to_publish.to_json will happen automatically
+  config = YAML.load_file("./config/messagebus.yml")
+  client = Messagebus::Client.new(config.merge(:logger => Logger.new("a/directory/to/my/file.log")))  
+  client.publish "jms.queue.testQueue1", object_to_publish
+
+=== Watch Tail Processing Log ===
+
+The tail will start displaying new log messages.
+
+  $ tail -f log/messagebus-client.log
+
+
+== Consuming messages ==
+You have 2 options. Option 1 is write your own consumer class. See [[https://github.com/groupon/Message-Bus/blob/master/mbus-ruby/examples/consumer_topic_example.rb|consumer_topic_example.rb]]
+for how to write your own consumer.
+
+Your other option is to use the Message Swarm code. Try examples/swarm_example.sh. See [[https://github.com/groupon/Message-Bus/wiki/Swarm-Consumer|Swarm Guide]] for more on that.
+
+== More detail needed? ==
+
+Contact us via email! dpe@groupon.com
+
+If something is not behaving intuitively, it is a bug and should be reported.
+Report it here: https://github.groupon.com/data-infrastructure/ruby-messagebus
+
+== Note on Patches/Pull Requests ==
+
+* Fork the project.
+* Make your feature addition or bug fix.
+* Commit, do not mess with rakefile, version, or history.
+* Send me a pull request. Bonus points for topic branches.
+
+== Copyright ==
+
+Copyright (c) 2011 Groupon. Released under the MIT License.
+
+See LICENSE for details.

data/Rakefile

@@ -0,0 +1,24 @@
+require "rubygems"
+require "rubygems/package_task"
+require "rspec/core/rake_task"
+
+SPEC = begin
+  file = File.expand_path(File.dirname(__FILE__) + '/messagebus.gemspec')
+  eval(File.read(file), binding, file)
+end
+
+Gem::PackageTask.new(SPEC) {}
+
+desc "Run all specs"
+RSpec::Core::RakeTask.new(:spec) do |t|
+  t.rspec_opts = %w[--color --format=progress]
+  t.pattern = 'spec/**/*_spec.rb'
+end
+
+desc "Run code coverage"
+RSpec::Core::RakeTask.new(:coverage) do |t|
+  t.rspec_opts = %w[--color --format=progress]
+  t.rcov = true
+  t.rcov_opts = %w[--exclude='gems,spec']
+  t.pattern = 'spec/**/*_spec.rb'
+end

data/bin/console

@@ -0,0 +1,11 @@
+#!/usr/bin/env ruby
+
+base = File.dirname(__FILE__)
+$:.unshift File.expand_path(File.join(base, "..", "lib"))
+
+require "irb"
+require "yaml"
+require "messagebus"
+
+IRB.start
+

data/bin/messagebus_swarm

@@ -0,0 +1,77 @@
+#!/usr/bin/env ruby
+
+require "optparse"
+
+require 'messagebus'
+require 'messagebus/swarm/controller'
+
+options = {
+  :requires => [],
+  :configuration_source => Messagebus::Swarm::Controller::ConfigurationSource.new(:default, nil)
+}
+
+parser = OptionParser.new do |opts|
+  opts.banner = "Usage: swarm [options] COMMAND"
+
+  opts.separator ""
+  opts.separator "Options:"
+
+  opts.on("--config_path PATH", "A path to a yaml configuration file") do |config_path|
+    options[:configuration_source] = Messagebus::Swarm::Controller::ConfigurationSource.new(:path, config_path)
+  end
+
+  opts.on("--config_eval RUBY_CODE", "Ruby that evaluates to a configuration hash. This is ran after any requires") do |config_eval|
+    options[:configuration_source] = Messagebus::Swarm::Controller::ConfigurationSource.new(:eval, config_eval)
+  end
+
+  opts.on("-h", "--help", "Show help message") do
+    puts opts
+  end
+
+  opts.on("-r", "--require FILE", "A file to require before starting the swarm. Repeatable.") do |file|
+    options[:requires] << file
+  end
+
+  opts.on("--pidfile PID_FILE", "A file to write the PID to") do |file|
+    options[:pidfile] = file
+  end
+
+  opts.separator ""
+  opts.separator "Commands:"
+  opts.separator "  start                   Start all workers in the configuration"
+  opts.separator "  start DESTINATION COUNT Start COUNT workers for DESTINATION swarm listeners"
+  opts.separator "  stop pid                Kill a swarm"
+  opts.separator ""
+end
+
+parser.parse!
+
+Messagebus::Swarm::Controller.require_files(options[:requires])
+if pidfile = options[:pidfile]
+  pid = Process.pid
+  Messagebus::Swarm::Controller.write_pid(pidfile)
+  at_exit do
+    # we need this check to make sure we only bother doing this if
+    # we are the parent process that's shutting down
+    # at_exit is carried down when forking
+    Messagebus::Swarm::Controller.delete_pid(pidfile) if Process.pid == pid
+  end
+end
+
+case ARGV[0]
+when 'start'
+  destination_name, drone_count = ARGV[1], ARGV[2]
+
+  log_file = options[:configuration_source].configuration_hash["worker_log_file"]
+  logger = log_file ? Logger.new(log_file) : Messagebus::Client.logger
+
+  Messagebus::Swarm::Controller.start(options[:configuration_source], logger, destination_name, drone_count)
+  exit(0)
+when 'stop'
+  pid = ARGV[1] or raise "Must specify a pid of the process to stop"
+  Messagebus::Swarm::Controller.stop(pid.to_i)
+  exit(0)
+else
+  puts parser.help
+  exit(1)
+end

data/lib/messagebus.rb

@@ -0,0 +1,62 @@
+# Copyright (c) 2012, Groupon, Inc.
+# All rights reserved.
+# 
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 
+# Redistributions of source code must retain the above copyright notice,
+# this list of conditions and the following disclaimer.
+# 
+# Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+# 
+# Neither the name of GROUPON nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+# 
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
+# IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
+# TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
+# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+$:.unshift(File.join(File.dirname(__FILE__), "..", "vendor/gems"))
+
+require "messagebus/version"
+require 'messagebus/logger'
+require 'messagebus/message'
+require "messagebus/messagebus_types"
+require 'messagebus/custom_errors'
+require 'messagebus/error_status'
+require 'messagebus/validations'
+require 'messagebus/connection'
+require 'messagebus/producer'
+require 'messagebus/consumer'
+require 'messagebus/cluster_map'
+require 'messagebus/client'
+
+require 'thrift'
+require 'base64'
+require 'stomp'
+
+module Messagebus
+  # These are deprecated and not used anymore
+  DEST_TYPE_QUEUE = 'queue'
+  DEST_TYPE_TOPIC = 'topic'
+
+  ACK_TYPE_AUTO_CLIENT = 'autoClient'
+  ACK_TYPE_CLIENT = 'client'
+
+  LOG_DEFAULT_FILE = '/tmp/messagebus_client.log'
+
+  SERVER_REGEX = /^[a-zA-Z0-9\_.-]+:[0-9]+$/
+end
+

data/lib/messagebus/client.rb

@@ -0,0 +1,166 @@
+# Copyright (c) 2012, Groupon, Inc.
+# All rights reserved.
+# 
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 
+# Redistributions of source code must retain the above copyright notice,
+# this list of conditions and the following disclaimer.
+# 
+# Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+# 
+# Neither the name of GROUPON nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+# 
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
+# IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
+# TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
+# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+require "messagebus/dottable_hash"
+require "benchmark"
+require "logger"
+
+module Messagebus
+  class Client
+    class InvalidDestinationError < StandardError; end
+    class InitializationError < StandardError; end
+
+    attr_reader :config, :logger, :cluster_map, :last_reload_time
+
+    def self.start(config_hash)
+      client = new(config_hash)
+      client.start
+
+      if block_given?
+        begin
+          yield client
+        ensure
+          client.stop
+        end
+      end
+      client
+    end
+
+    def initialize(config_hash)
+
+      @@logger = if provided_logger = config_hash.delete(:logger)
+        provided_logger
+      elsif log_file = config_hash["log_file"]
+        Logger.new(log_file)
+      else
+        Logger.new(Messagebus::LOG_DEFAULT_FILE)
+      end
+
+      # This is required to do a deep clone of config hash object
+      @config = DottableHash.new(Marshal.load(Marshal.dump(config_hash)))
+      @config.merge!(@config.cluster_defaults) if @config.cluster_defaults
+      @@logger.level = Logger::Severity.const_get(@config.log_level.upcase) if @config.log_level
+
+      @enable_client_logger_thread_debugging = config.enable_client_logger_thread_debugging
+
+      logger.debug "Initializing Messagebus client."
+      @cluster_map = ClusterMap.new(@config)
+      #added for reloading config on interval
+      @last_reload_time = Time.now
+    end
+
+    # Starts up all the connections to the bus.
+    # Optionally takes a block to which it yields self. When the block is
+    # passed, it will auto close the connections after the block finishes.
+    def start
+      if @config.enable_auto_init_connections
+        logger.info "auto enable connections set, starting clusters."
+        @cluster_map.start
+      else
+        logger.info "Config['enable_auto_init_connections'] is false, will not start any messagebus producers."
+      end
+    end
+
+    def stop
+      cluster_map.stop
+    end
+
+    def logger
+      @@logger ||= @config['log_file'] ? Logger.new(@config['log_file']) : Logger.new(Messagebus::LOG_DEFAULT_FILE)
+    end
+
+    def publish(destination_name, object, delay_ms = 0, safe = true, binary = false, headers = {})
+
+      if !(@config.enable_auto_init_connections)
+        logger.warn "Config['enable_auto_init_connections'] is false, not publishing destination_name=#{destination_name}, message_contents=#{object.inspect}"
+        false
+      else
+        producer = cluster_map.find(destination_name)
+        if producer.nil?
+          logger.error "Not publishing due to unconfigured destionation name. destination_name=#{destination_name}, message=#{object.inspect}"
+          raise InvalidDestinationError, "Destination #{destination_name} not found"
+        end
+
+        if binary
+          message = Messagebus::Message.create(object, nil, binary)
+        else
+          message = Messagebus::Message.create(object)
+        end
+
+        logger.info "Publishing to destination_name=#{destination_name}, message_id=#{message.message_id}, message_contents=#{object.inspect}"
+
+        begin
+          publish_result = nil
+          duration = Benchmark.realtime do
+            publish_result = producer.publish(destination_name, message, headers(delay_ms).merge(headers), safe)
+          end
+          duration = (duration * 1_000).round
+
+          if publish_result
+            logger.info "Message publishing to #{destination_name} took #{duration} result=success destination_name=#{destination_name}, message_id=#{message.message_id}, duration=#{duration}ms"
+            true
+          else
+            logger.error "Failed to publish message result=fail destination_name=#{destination_name}, message_id=#{message.message_id}, duration=#{duration}ms, message_contents=#{object.inspect}"
+            false
+          end
+        rescue => e
+          logger.error "Failed to publish message result=error destination_name=#{destination_name}, message_id=#{message.message_id}, duration=#{duration}ms, message_contents=#{object.inspect}, error=#{e.inspect}, backtrace=#{e.backtrace.join("|")}"
+          false
+        end
+      end
+    end
+    
+    def reload_config_on_interval(config, interval = 300)
+      logger.info "Relaoding client configs after interval=#{interval}"
+      now = Time.now
+      if(now - @last_reload_time) >= interval
+        @cluster_map.update_config(config)
+        @last_reload_time = now
+      end
+    end
+    
+    def headers(delay)
+      headers = {}
+      unless delay == 0
+        schedule_time = (Time.now.to_i * 1000 + delay).to_s
+        headers.merge!({
+          Messagebus::Producer::SCHEDULED_DELIVERY_TIME_MS_HEADER => schedule_time
+        })
+      end
+      headers
+    end
+
+    class << self
+      def logger
+        @@logger ||= Logger.new(Messagebus::LOG_DEFAULT_FILE)
+      end
+    end
+  end
+end