hutch 0.10.0 → 0.11.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 803369cb7619eae888379612cf278d49133a3f68
-  data.tar.gz: 4eb4a16d6b7a12441b9e56d972ebcc413b5fec82
+  metadata.gz: b4ae8b34c445db665d717e7c4b935ae246c793c4
+  data.tar.gz: d92e37a658a7a46a08500cf7c14589f7f6df2a17
 SHA512:
-  metadata.gz: 3f557376e9db220197b6f6f99cd28621593ed3a5f0bc96e26c471fb40b7b54e3f79564cad661873b223c7688713112bbf5536b705efdb0d3aa4b86d88725cd6b
-  data.tar.gz: d497420735d0567a167290a9de4130d66635f4ef36d5b1f674ca77e0320956518014cbc91054d72b10dc16ef3c2edc5498e47dd7a1b6ddce4edb013a757158ff
+  metadata.gz: 149952fbd91217d9bb7a92c7eac65607b45af0b35aaf0bba3424996376fa0f49fd205efc02b67e5b6a7f7a7a66a1dd22ad0e35ad576772905b0d91dc9da62950
+  data.tar.gz: 6a128089af08c252778bf0cef44cfbbc522aea4938c7a4904c9ea74e21528fc13559a3ecb5011700c36a4e87bdd9b98e171593a3e28829312d3b9162c9774d7e

data/CHANGELOG.md

@@ -1,14 +1,73 @@
-## 0.10.0 — unreleased
+## 0.11.0 — unreleased
+
+### Publisher Confirms Support
+
+`:force_publisher_confirms` is a new configuration option that forces `Hutch.publish` to wait
+for a confirm for every message published. Note that this **will cause a significant drop in throughput**.
+
+`Hutch::Broker#confirm_select` and `Hutch::Broker#wait_for_confirms` are new public API methods
+that delegate to their respective `Bunny::Channel` counterparts. `Hutch::Broker#confirm_select`
+can be used to handle confirms with a callback instead of waiting:
+
+``` ruby
+broker.confirm_select do |delivery_tag, multiple, nack|
+  # ...
+end
+```
+
 
 ### Bunny Update
 
-Bunny is updated to `1.4.x`.
+Bunny is updated to [1.6.0](http://blog.rubyrabbitmq.info/blog/2014/10/31/bunny-1-dot-6-0-is-released/).
+
+
+## 0.10.0 — Oct 22, 2014
+
+### Configuration via URI
+
+Hutch now supports a new configuration key, `:uri`, which allows
+connection configuration via a URI.
+
+Note that since the URI has to include credentials, this option
+is not available on the command line.
+
+### Bunny Update
+
+Bunny is updated to `1.5.1`, which mitigates the POODLE attack
+by disabling SSL 3.0 where possible.
+
+### Payload in Error Handlers
+
+Error handlers will now have access to message payload.
+
+Contributed by Daniel Farrell.
 
 ### Exceptions in Error Handlers Don't Prevent Nacks
 
 Exceptions in error handlers no longer prevent messages from being
 `basic.nack`-ed.
 
+### Pid File Support
+
+`:pidfile` is a new configuration option that stores Hutch process
+PID in a file at provided path.
+
+Contributed by Rustam Sharshenov.
+
+### More Info on Message
+
+Bunny's `delivery_info`, `properties` and payload are now accessible on `Hutch::Message`.
+
+Contributed by gregory.
+
+
+### Optional Config Parameters
+
+`Hutch::Config` constructor now accepts an extra hash of optional
+configuration parameters.
+
+Contributed by Ignazio Mostallino.
+
 
 ## 0.9.0 — May 13, 2014
 

data/README.md

@@ -26,6 +26,10 @@ over RabbitMQ. Hutch is opinionated: it uses topic exchanges for message
 distribution and makes some assumptions about how consumers and publishers
 should work.
 
+With Hutch, consumers are stored in separate files and include the `Hutch::Consumer` module.
+They are then loaded by a command line runner which connects to RabbitMQ, sets up queues and bindings,
+and so on. Publishers connect to RabbitMQ via `Hutch.connect` and publish using `Hutch.publish`.
+
 Hutch uses [Bunny](http://rubybunny.info) under the hood.
 
 
@@ -160,6 +164,26 @@ Hutch.connect
 Hutch.publish('routing.key', subject: 'payment', action: 'received')
 ```
 
+### Producer Configuration
+
+Producers are not run with the 'hutch' command. You can specify configuration
+options as follows:
+
+```ruby
+Hutch::Config.set(:mq_exchange, 'name')
+```
+
+### Publisher Confirms
+
+For maximum message reliability when producing messages, you can force Hutch to use
+[Publisher Confirms](https://www.rabbitmq.com/confirms.html) and wait for a confirmation
+after every message published. This is the safest possible option for publishers
+but also results in a **significant throughput drop**.
+
+```ruby
+Hutch::Config.set(:force_publisher_confirms, true)
+```
+
 ### Writing Well-Behaved Publishers
 
 You may need to send messages to Hutch from languages other than Ruby. This
@@ -174,11 +198,8 @@ send messages to Hutch.
 - Use message routing keys that match those used in your Hutch consumers.
 - Be sure your exchanges are marked as durable. In the Ruby AMQP gem, this is
   done by passing `durable: true` to the exchange creation method.
-- Mark your messages as persistent. This is done by passing `persistent: true`
-  to the publish method in Ruby AMQP.
-- Wrapping publishing code in transactions or using publisher confirms is
-  highly recommended. This can be slightly tricky, see [this issue][pc-issue]
-  and [this gist][pc-gist] for more info.
+- Publish messages as persistent.
+- Using publisher confirms is highly recommended.
 
 Here's an example of a well-behaved publisher, minus publisher confirms:
 
@@ -192,15 +213,42 @@ AMQP.connect(host: config[:host]) do |connection|
 end
 ```
 
+If using publisher confirms with amqp gem, see [this issue][pc-issue]
+and [this gist][pc-gist] for more info.
+
+## Configuration Reference
+
+### Config File
+
+It is recommended to use a separate config file, unless you use URIs for connection (see below).
+
+Known configuration parameters are:
+
+ * `mq_host`: RabbitMQ hostname (default: `localhost`)
+ * `mq_port`: RabbitMQ port (default: `5672`)
+ * `mq_vhost`: vhost to use (default: `/`)
+ * `mq_username`: username to use (default: `guest`, only can connect from localhost as of RabbitMQ 3.3.0)
+ * `mq_password`: password to use (default: `guest`)
+ * `mq_tls`: should TLS be used? (default: `false`)
+ * `mq_tls_cert`: path to client TLS certificate (public key)
+ * `mq_tls_key`: path to client TLS private key
+ * `require_paths`: array of paths to require
+ * `autoload_rails`: should Hutch command line runner try to automatically load Rails environment files?
+ * `daemonise`: should Hutch runner process daemonise?
+ * `pidfile`: path to PID file the runner should use
+ * `channel_prefetch`: basic.qos prefetch value to use (default: `0`, no limit). See Bunny and RabbitMQ documentation.
+ * `publisher_confirms`: enables publisher confirms. Leaves it up to the app how they are
+                         tracked (e.g. using `Hutch::Broker#confirm_select` callback or `Hutch::Broker#wait_for_confirms`)
+ * `force_publisher_confirms`: enables publisher confirms, forces `Hutch::Broker#wait_for_confirms` for every publish. **This is the safest option which also offers the lowest throughput**.
+ * `log_level`: log level used by the standard Ruby logger (default: `Logger::INFO`)
+ * `mq_exchange`: exchange to use for publishing (default: `hutch`)
+
+
 ## Supported RabbitMQ Versions
 
 Hutch requires RabbitMQ 2.x or later. 3.x releases
 are recommended.
 
-
-[pc-issue]: https://github.com/ruby-amqp/amqp/issues/92
-[pc-gist]: https://gist.github.com/3042381
-
 ---
 
 GoCardless ♥ open source. If you do too, come [join us](https://gocardless.com/jobs/backend_developer).

data/hutch.gemspec

@@ -1,7 +1,7 @@
 require File.expand_path('../lib/hutch/version', __FILE__)
 
 Gem::Specification.new do |gem|
-  gem.add_runtime_dependency 'bunny', '>= 1.5.1'
+  gem.add_runtime_dependency 'bunny', '>= 1.6.1'
   gem.add_runtime_dependency 'carrot-top', '~> 0.0.7'
   gem.add_runtime_dependency 'multi_json', '~> 1.5'
   gem.add_development_dependency 'rspec', '~> 3.0'

data/lib/hutch.rb

@@ -37,6 +37,13 @@ module Hutch
     end
   end
 
+  def self.disconnect
+    if @broker
+      @broker.disconnect
+      @connected = false
+    end
+  end
+
   def self.broker
     @broker
   end

data/lib/hutch/broker.rb

@@ -88,6 +88,10 @@ module Hutch
       logger.info 'opening rabbitmq channel'
       @channel = connection.create_channel.tap do |ch|
         ch.prefetch(@config[:channel_prefetch]) if @config[:channel_prefetch]
+        if @config[:publisher_confirms] || @config[:force_publisher_confirms]
+          logger.info 'enabling publisher confirms'
+          ch.confirm_select
+        end
       end
     end
 
@@ -188,10 +192,26 @@ module Hutch
       properties[:message_id] ||= generate_id
 
       logger.info("publishing message '#{message.inspect}' to #{routing_key}")
-      @exchange.publish(JSON.dump(message), {persistent: true}.
+
+      response = @exchange.publish(JSON.dump(message), {persistent: true}.
         merge(properties).
         merge(global_properties).
         merge(non_overridable_properties))
+
+      channel.wait_for_confirms if @config[:force_publisher_confirms]
+      response
+    end
+
+    def confirm_select(*args)
+      @channel.confirm_select(*args)
+    end
+
+    def wait_for_confirms
+      @channel.wait_for_confirms
+    end
+
+    def using_publisher_confirmations?
+      @channel.using_publisher_confirmations?
     end
 
     private

data/lib/hutch/config.rb

@@ -31,7 +31,13 @@ module Hutch
         namespace: nil,
         daemonise: false,
         pidfile: nil,
-        channel_prefetch: 0
+        channel_prefetch: 0,
+        # enables publisher confirms, leaves it up to the app
+        # how they are tracked
+        publisher_confirms: false,
+        # like `publisher_confirms` above but also
+        # forces waiting for a confirm for every publish
+        force_publisher_confirms: false
       }.merge(params)
     end
 
@@ -61,6 +67,10 @@ module Hutch
       @config
     end
 
+    def self.to_hash
+      self.user_config
+    end
+
     def self.load_from_file(file)
       YAML.load(file).each do |attr, value|
         Hutch::Config.send("#{attr}=", value)

data/lib/hutch/version.rb

@@ -1,4 +1,4 @@
 module Hutch
-  VERSION = '0.10.0'.freeze
+  VERSION = '0.11.0'.freeze
 end
 

data/lib/hutch/worker.rb

@@ -65,7 +65,7 @@ module Hutch
       queue = @broker.queue(consumer.get_queue_name)
       @broker.bind_queue(queue, consumer.routing_keys)
 
-      queue.subscribe(ack: true) do |delivery_info, properties, payload|
+      queue.subscribe(manual_ack: true) do |delivery_info, properties, payload|
         handle_message(consumer, delivery_info, properties, payload)
       end
     end

data/spec/hutch/broker_spec.rb

@@ -92,6 +92,18 @@ describe Hutch::Broker do
         broker.set_up_amqp_connection
       end
     end
+
+    context 'with force_publisher_confirms set' do
+      let(:force_publisher_confirms_value) { true }
+      before { config[:force_publisher_confirms] = force_publisher_confirms_value }
+      after  { broker.disconnect }
+
+      it 'waits for confirmation' do
+        expect_any_instance_of(Bunny::Channel).
+          to receive(:confirm_select)
+        broker.set_up_amqp_connection
+      end
+    end
   end
 
   describe '#set_up_api_connection', rabbitmq: true do
@@ -252,6 +264,28 @@ describe Hutch::Broker do
           end
         end
       end
+
+      context 'with force_publisher_confirms not set in the config' do
+        it 'does not wait for confirms on the channel' do
+          expect_any_instance_of(Bunny::Channel).
+            to_not receive(:wait_for_confirms)
+          broker.publish('test.key', 'message')
+        end
+      end
+
+      context 'with force_publisher_confirms set in the config' do
+        let(:force_publisher_confirms_value) { true }
+
+        before do
+          config[:force_publisher_confirms] = force_publisher_confirms_value
+        end
+
+        it 'waits for confirms on the channel' do
+          expect_any_instance_of(Bunny::Channel).
+            to receive(:wait_for_confirms)
+          broker.publish('test.key', 'message')
+        end
+      end
     end
 
     context 'without a valid connection' do

data/spec/hutch/worker_spec.rb

@@ -33,7 +33,7 @@ describe Hutch::Worker do
     end
 
     it 'sets up a subscription' do
-      expect(queue).to receive(:subscribe).with(ack: true)
+      expect(queue).to receive(:subscribe).with(manual_ack: true)
       worker.setup_queue(consumer)
     end
   end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: hutch
 version: !ruby/object:Gem::Version
-  version: 0.10.0
+  version: 0.11.0
 platform: ruby
 authors:
 - Harry Marr
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-10-22 00:00:00.000000000 Z
+date: 2014-11-14 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bunny
@@ -16,14 +16,14 @@ dependencies:
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
-        version: 1.5.1
+        version: 1.6.1
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
-        version: 1.5.1
+        version: 1.6.1
 - !ruby/object:Gem::Dependency
   name: carrot-top
   requirement: !ruby/object:Gem::Requirement