hutch 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 41727b6727cb22516abf3d78004b4b079d7f4227
+  data.tar.gz: 7d6d4f335003d299b94c33c4bf297c8912932cbb
+SHA512:
+  metadata.gz: c7e22f63f21e5afbd94f503b4933b0f43522625e592fb1f2be357ca2728c7bba378db1761bf44690c5c0cf318fb898d6cfaae1eeb7870c964638fc73a78c3b00
+  data.tar.gz: 007464c716f89732b2ca924223e90d92db730f5ebb23b740efb8acbc9b7a00e4285d41f4fb50af4435078fd855d77ee0fd820ccce328294556887ce608759fe7

data/.gitignore

@@ -0,0 +1,2 @@
+.bundle
+hutch-*.gem

data/Gemfile

@@ -0,0 +1,18 @@
+source 'https://rubygems.org'
+
+gemspec
+
+group :development do
+  gem "rake"
+  gem "guard", "~> 0.8.8"
+  gem "guard-rspec", "~> 0.5.4"
+end
+
+group :development, :test do
+  gem "sentry-raven"
+end
+
+group :development, :darwin do
+  gem "rb-fsevent", "~> 0.9"
+  gem "growl", "~> 1.0.3"
+end

data/Guardfile

@@ -0,0 +1,5 @@
+guard 'rspec', :version => 2, :cli => '--color --format doc' do
+  watch(%r{^spec/.+_spec\.rb$})
+  watch(%r{^lib/(.+)\.rb$}) { |m| "spec/#{m[1]}_spec.rb" }
+  watch('spec/spec_helper.rb') { "spec" }
+end

data/README.md

@@ -0,0 +1,136 @@
+![](http://cl.ly/image/3h0q3F3G142K/hutch.png)
+
+Hutch is a Ruby library for enabling asynchronous inter-service communication
+in a service-oriented architecture, using RabbitMQ.
+
+
+## Defining Consumers
+
+Consumers receive messages from a RabbitMQ queue. That queue may be bound to
+one or more topics (represented by routing keys).
+
+To create a consumer, include the `Hutch::Consumer` module in a class that
+defines a `#process` method. `#process` should take a single argument, which
+will be a `Message` object. The `Message` object encapsulates the message data,
+along with any associated metadata. To access properties of the message, use
+Hash-style indexing syntax:
+
+```ruby
+message[:id]  # => "02ABCXYZ"
+```
+
+To subscribe to a topic, pass a routing key to `consume` in the class
+definition. To bind to multiple routing keys, simply pass extra routing keys
+in as additional arguments. Refer to the [RabbitMQ docs on topic exchanges
+][topic-docs] for more information about how to use routing keys. Here's an
+example consumer:
+
+```ruby
+class FailedPaymentConsumer
+  include Hutch::Consumer
+  consume 'gc.ps.payment.failed'
+
+  def process(message)
+    mark_payment_as_failed(message[:id])
+  end
+end
+```
+
+If you are using Hutch with Rails and want to make Hutch log to the Rails
+logger rather than `stdout`, add this to `config/initializers/hutch.rb`
+
+```ruby
+Hutch::Logging.logger = Rails.logger
+```
+
+[topic-docs]: http://www.rabbitmq.com/tutorials/tutorial-five-python.html
+
+
+## Running Hutch
+
+After installing the Hutch gem, you should be able to start it by simply
+running `hutch` on the command line. `hutch` takes a number of options:
+
+```console
+$ hutch -h
+usage: hutch [options]
+        --mq-host HOST               Set the RabbitMQ host
+        --mq-port PORT               Set the RabbitMQ port
+        --mq-exchange EXCHANGE       Set the RabbitMQ exchange
+        --mq-vhost VHOST             Set the RabbitMQ vhost
+        --mq-api-host HOST           Set the RabbitMQ API host
+        --mq-api-port PORT           Set the RabbitMQ API port
+        --mq-api-username USERNAME   Set the RabbitMQ API username
+        --mq-api-password PASSWORD   Set the RabbitMQ API password
+        --require PATH               Require a Rails app or path
+    -q, --quiet                      Quiet logging
+    -v, --verbose                    Verbose logging
+        --version                    Print the version and exit
+    -h, --help                       Show this message and exit
+```
+
+The first three are for configuring which RabbitMQ instance to connect to.
+`--require` is covered in the next section. The remainder are self-explanatory.
+
+### Loading Consumers
+
+Using Hutch with a Rails app is simple. Either start Hutch in the working
+directory of a Rails app, or pass the path to a Rails app in with the
+`--require` option. Consumers defined in Rails apps should be placed with in
+the `app/consumers/` directory, to allow them to be auto-loaded when Rails
+boots.
+
+To require files that define consumers manually, simply pass each file as an
+option to `--require`. Hutch will automatically detect whether you've provided
+a Rails app or a standard file, and take the appropriate behaviour:
+
+```bash
+$ hutch --require path/to/rails-app  # loads a rails app
+$ hutch --require path/to/file.rb    # loads a ruby file
+```
+
+## Producers
+
+Hutch includes a `publish` method for sending messages to Hutch consumers. When
+possible, this should be used, rather than directly interfacing with RabbitMQ
+libraries.
+
+```ruby
+Hutch.publish('routing.key', subject: 'payment', action: 'received')
+```
+
+### Writing Well-Behaved Publishers
+
+You may need to send messages to Hutch from languages other than Ruby. This
+prevents the use of `Hutch.publish`, requiring custom publication code to be
+written. There are a few things to keep in mind when writing producers that
+send messages to Hutch.
+
+- Make sure that the producer exchange name matches the exchange name that
+  Hutch is using.
+- Hutch works with topic exchanges, check the producer is also using topic
+  exchanges.
+- Use message routing keys that match those used in your Hutch consumers.
+- Be sure your exchanges are marked as durable. In the Ruby AMQP gem, this is
+  done by passing `durable: true` to the exchange creation method.
+- Mark your messages as persistent. This is done by passing `persistent: true`
+  to the publish method in Ruby AMQP.
+- Wrapping publishing code in transactions or using publisher confirms is
+  highly recommended. This can be slightly tricky, see [this issue][pc-issue]
+  and [this gist][pc-gist] for more info.
+
+Here's an example of a well-behaved publisher, minus publisher confirms:
+
+```ruby
+AMQP.connect(host: config[:host]) do |connection|
+  channel  = AMQP::Channel.new(connection)
+  exchange = channel.topic(config[:exchange], durable: true)
+
+  message = JSON.dump({ subject: 'Test', id: 'abc' })
+  exchange.publish(message, routing_key: 'test', persistent: true)
+end
+```
+
+[pc-issue]: https://github.com/ruby-amqp/amqp/issues/92
+[pc-gist]: https://gist.github.com/3042381
+

data/Rakefile

@@ -0,0 +1,14 @@
+require 'rspec/core/rake_task'
+
+desc "Run an IRB session with Hutch pre-loaded"
+task :console do
+  exec "irb -I lib -r hutch"
+end
+
+desc "Run the test suite"
+RSpec::Core::RakeTask.new(:spec) do |t|
+  t.pattern = FileList['spec/**/*_spec.rb']
+  t.rspec_opts = %w(--color --format doc)
+end
+
+task :default => :spec

data/bin/hutch

@@ -0,0 +1,8 @@
+#!/usr/bin/env ruby
+
+require_relative '../lib/hutch'
+require_relative '../lib/hutch/cli'
+
+cli = Hutch::CLI.new
+cli.run
+

data/circle.yml

@@ -0,0 +1,3 @@
+machine:
+  services:
+    - rabbitmq-server

data/examples/consumer.rb

@@ -0,0 +1,13 @@
+require 'hutch'
+
+class TestConsumer
+  include Hutch::Consumer
+  consume 'hutch.test'
+
+  def process(message)
+    puts "TestConsumer got a message: #{message}"
+    puts "Processing..."
+    sleep(1)
+    puts "Done"
+  end
+end

data/examples/producer.rb

@@ -0,0 +1,10 @@
+require 'hutch'
+
+Hutch.connect
+loop do
+  print "Press return to send test message..."
+  gets
+  Hutch.publish('hutch.test', subject: 'test message')
+  puts "Send message with routing key 'hutch.test'"
+end
+

data/hutch.gemspec

@@ -0,0 +1,22 @@
+require File.expand_path('../lib/hutch/version', __FILE__)
+
+Gem::Specification.new do |gem|
+  gem.add_runtime_dependency 'bunny', '0.10.6'
+  gem.add_runtime_dependency 'carrot-top', '~> 0.0.7'
+  gem.add_runtime_dependency 'multi_json', '~> 1.5'
+  gem.add_development_dependency 'rspec', '~> 2.12.0'
+
+  gem.name = 'hutch'
+  gem.summary = 'Easy inter-service communication using RabbitMQ.'
+  gem.description = 'Hutch is a Ruby library for enabling asynchronous ' +
+                    'inter-service communication using RabbitMQ.'
+  gem.version = Hutch::VERSION.dup
+  gem.authors = ['Harry Marr']
+  gem.email = ['developers@gocardless.com']
+  gem.homepage = 'https://github.com/gocardless/hutch'
+  gem.require_paths = ['lib']
+  gem.license = 'MIT'
+  gem.executables = ['hutch']
+  gem.files = `git ls-files`.split("\n")
+  gem.test_files = `git ls-files -- spec/*`.split("\n")
+end

data/lib/hutch.rb

@@ -0,0 +1,40 @@
+module Hutch
+  require 'hutch/consumer'
+  require 'hutch/worker'
+  require 'hutch/logging'
+  require 'hutch/error_handlers/logger'
+  ErrorHandlers.autoload :Sentry, 'hutch/error_handlers/sentry'
+
+  def self.register_consumer(consumer)
+    self.consumers << consumer
+  end
+
+  def self.consumers
+    @consumers ||= []
+  end
+
+  def self.logger
+    Hutch::Logging.logger
+  end
+
+  def self.connect(config = Hutch::Config)
+    unless connected?
+      @broker = Hutch::Broker.new(config)
+      @broker.connect
+      @connected = true
+    end
+  end
+
+  def self.broker
+    @broker
+  end
+
+  def self.connected?
+    @connected
+  end
+
+  def self.publish(routing_key, message)
+    @broker.publish(routing_key, message)
+  end
+end
+

data/lib/hutch/broker.rb

@@ -0,0 +1,175 @@
+require 'bunny'
+require 'carrot-top'
+require 'securerandom'
+require 'hutch/logging'
+require 'hutch/exceptions'
+
+module Hutch
+  class Broker
+    include Logging
+
+    attr_accessor :connection, :channel, :exchange, :api_client
+
+    def initialize(config = nil)
+      @config = config || Hutch::Config
+    end
+
+    def connect
+      set_up_amqp_connection
+      set_up_api_connection
+
+      if block_given?
+        yield
+        disconnect
+      end
+    end
+
+    def disconnect
+      @channel.close    if @channel
+      @connection.close if @connection
+      @channel, @connection, @exchange, @api_client = nil, nil, nil, nil
+    end
+
+    # Connect to RabbitMQ via AMQP. This sets up the main connection and
+    # channel we use for talking to RabbitMQ. It also ensures the existance of
+    # the exchange we'll be using.
+    def set_up_amqp_connection
+      host, port, vhost = @config[:mq_host], @config[:mq_port]
+      username, password = @config[:mq_username], @config[:mq_password]
+      vhost = @config[:mq_vhost]
+      uri = "#{username}:#{password}@#{host}:#{port}/#{vhost.sub(/^\//, '')}"
+      logger.info "connecting to rabbitmq (amqp://#{uri})"
+
+      @connection = Bunny.new(host: host, port: port, vhost: vhost,
+                              username: username, password: password,
+                              heartbeat: 1, automatically_recover: true,
+                              network_recovery_interval: 1)
+      @connection.start
+
+      logger.info 'opening rabbitmq channel'
+      @channel = @connection.create_channel
+
+      exchange_name = @config[:mq_exchange]
+      logger.info "using topic exchange '#{exchange_name}'"
+      @exchange = @channel.topic(exchange_name, durable: true)
+    rescue Bunny::TCPConnectionFailed => ex
+      logger.error "amqp connection error: #{ex.message.downcase}"
+      uri = "amqp://#{host}:#{port}"
+      raise ConnectionError.new("couldn't connect to rabbitmq at #{uri}")
+    rescue Bunny::PreconditionFailed => ex
+      logger.error ex.message
+      raise WorkerSetupError.new('could not create exchange due to a type ' +
+                                 'conflict with an existing exchange, ' +
+                                 'remove the existing exchange and try again')
+    end
+
+    # Set up the connection to the RabbitMQ management API. Unfortunately, this
+    # is necessary to do a few things that are impossible over AMQP. E.g.
+    # listing queues and bindings.
+    def set_up_api_connection
+      host, port = @config[:mq_api_host], @config[:mq_api_port]
+      username, password = @config[:mq_username], @config[:mq_password]
+
+      management_uri = "http://#{username}:#{password}@#{host}:#{port}/"
+      logger.info "connecting to rabbitmq management api (#{management_uri})"
+
+      @api_client = CarrotTop.new(host: host, port: port,
+                                  user: username, password: password)
+      @api_client.exchanges
+    rescue Errno::ECONNREFUSED => ex
+      logger.error "api connection error: #{ex.message.downcase}"
+      raise ConnectionError.new("couldn't connect to api at #{management_uri}")
+    rescue Net::HTTPServerException => ex
+      logger.error "api connection error: #{ex.message.downcase}"
+      if ex.response.code == '401'
+        raise AuthenticationError.new('invalid api credentials')
+      else
+        raise
+      end
+    end
+
+    # Create / get a durable queue.
+    def queue(name)
+      @channel.queue(name, durable: true)
+    end
+
+    # Return a mapping of queue names to the routing keys they're bound to.
+    def bindings
+      results = Hash.new { |hash, key| hash[key] = [] }
+      @api_client.bindings.each do |binding|
+        next if binding['destination'] == binding['routing_key']
+        next unless binding['source'] == @config[:mq_exchange]
+        next unless binding['vhost'] == @config[:mq_vhost]
+        results[binding['destination']] << binding['routing_key']
+      end
+      results
+    end
+
+    # Bind a queue to the broker's exchange on the routing keys provided. Any
+    # existing bindings on the queue that aren't present in the array of
+    # routing keys will be unbound.
+    def bind_queue(queue, routing_keys)
+      # Find the existing bindings, and unbind any redundant bindings
+      queue_bindings = bindings.select { |dest, keys| dest == queue.name }
+      queue_bindings.each do |dest, keys|
+        keys.reject { |key| routing_keys.include?(key) }.each do |key|
+          logger.debug "removing redundant binding #{queue.name} <--> #{key}"
+          queue.unbind(@exchange, routing_key: key)
+        end
+      end
+
+      # Ensure all the desired bindings are present
+      routing_keys.each do |routing_key|
+        logger.debug "creating binding #{queue.name} <--> #{routing_key}"
+        queue.bind(@exchange, routing_key: routing_key)
+      end
+    end
+
+    # Each subscriber is run in a thread. This calls Thread#join on each of the
+    # subscriber threads.
+    def wait_on_threads(timeout)
+      # HACK: work_pool.join doesn't allow a timeout to be passed in, so we
+      #       use instance_variable_get to gain access to the threadpool, and
+      #       manuall call thread.join with a timeout
+      threads = work_pool_threads
+
+      # Thread#join returns nil when the timeout is hit. If any return nil,
+      # the threads didn't all join so we return false.
+      per_thread_timeout = timeout.to_f / threads.length
+      threads.none? { |thread| thread.join(per_thread_timeout).nil? }
+    end
+
+    def stop
+      @channel.work_pool.kill
+    end
+
+    def ack(delivery_tag)
+      @channel.ack(delivery_tag, false)
+    end
+
+    def publish(routing_key, message)
+      payload = JSON.dump(message)
+
+      if @connection && @connection.open?
+        logger.info "publishing message '#{message.inspect}' to #{routing_key}"
+        @exchange.publish(payload, routing_key: routing_key, persistent: true,
+                          timestamp: Time.now.to_i, message_id: generate_id)
+      else
+        logger.error "Unable to publish : routing key: #{routing_key}, " +
+                     "message: #{message}"
+      end
+    end
+
+    private
+
+    def work_pool_threads
+      # TODO: fix bunny so we don't need to do this
+      @channel.work_pool.instance_variable_get(:@threads)
+    end
+
+    def generate_id
+      SecureRandom.uuid
+    end
+  end
+end
+