mercury_amqp 0.4.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 459b852ab72eefefec630bac89fde701f7fc2018
-  data.tar.gz: a0ae43b454314876e4dc1be0e6ccf9480d8a9ed6
+  metadata.gz: 43d54035bd40df4089510aa9839d554f2e715c8f
+  data.tar.gz: c8e309676dbdb1f4488a23c95a060ca4e2d78f3c
 SHA512:
-  metadata.gz: 296ef00ec39d0419c4f55b185528a4d0ed905d236014ad0f8a432a310a01033cf4105c6c2c19a82d02cec850c4de3e8af2a96eba8f0e6d8f4761002a1ef971df
-  data.tar.gz: 7fcbe0aa189f7d5924312210ddaf7eb1fb6111ff0f4f74e38fddc1a7410f26a8820fd871cac8daaf0c93b363bee153cb61b1b85ca4ee760bc8c6240048758ca3
+  metadata.gz: c11bd808a2634847b2a7c1b97975a7c3f2601086a82c2b43ba4fc866aea0a068cae96d4d7ce5cd05a3c7678ff8beb69d17a1cc311a16f9c634ba0780184ef917
+  data.tar.gz: 0d458c16c5ea7706e56ec0d39549bb6cb762989b3c057c9426f5e401bccd05974b49f567cbdf9ec30d44c06d91caeca91d23d221d0074e59d8635b3c65b3e150

data/README.md

@@ -81,13 +81,17 @@ Indicates message handling succeeded. The message is removed from the queue.
 **nack**(_msg_)
 
 Indicates message handling failed, with the assumption that it might succeed at
-a later time. The message is returned to the queue.
+a later time. The message is returned to the front of the queue.
 
 **reject**(_msg_)
 
 Indicates message handling failed, with the assumption that it can never succeed.
 The message is removed from the queue.
 
+**republish**(_msg_)
+
+Like **nack**, except the message is returned to the _back_ of the queue.
+
 _Note:_ All operations create the referenced constructs if they do not already exist.
 
 ### Serialization
@@ -200,6 +204,212 @@ seql do
 end
 ```
 
+Monadic Interface
+-----------------
+
+The _monad_ is a versatile design pattern. There is plenty of
+literature online, but for now all you need to know is that mercury
+uses monad principles to chain together asynchronous operations. It
+all starts with a `Cps` object.
+
+```
+2.2.2 :005 > add1 = Cps.new { |n, &k| k.call(n+1) }
+ => #<Mercury::Cps:...>
+```
+
+Much like `Proc.new`, `Cps.new` merely captures some operation in an
+object but does not do any actual work. The key difference is that
+`Cps` captures a "continuation-passing style" ("CPS") operation. That is,
+instead of returning a value to the caller, the operation takes its
+continuation as an additional argument `k`. (The actual name doesn't matter,
+of course, but `k` is traditional.) `k` is simply a `Proc`.
+You can loosely think of it as the "return _`Proc`_", as opposed to
+the usual return _statement_.
+
+To invoke a `Proc` we call `Proc#call`. To invoke a `Cps`, we call
+`Cps#run`, passing the normal arguments as well as the continuation
+(the block).
+
+```
+2.2.2 :006 > add1.run(2) { |result| puts "result = #{result}" }
+result = 3
+```
+
+As you've seen already, asynchronous APIs are closely tied to CPS. In
+the case of mercury, an operation may involve a conversation with the
+server. CPS allows our code to go off and do other things -- namely,
+handle other independent requests -- but also be notified when the
+operation finally completes.
+
+### Combining operations
+
+`Cps` provides a means of combining operations into a larger
+operation: `Cps#and_then`.
+
+```ruby
+def add1(n)
+  Cps.new { |&k| k.call(n+1) }
+end
+
+def print_value(n)
+  Cps.new do |&k|
+    puts "value = #{n}"
+    k.call
+  end
+end
+
+def add1_and_print(n)
+  add1(n).and_then { |v| print_value(v) }
+end
+```
+```
+2.2.2 :028 > add1_and_print(2).run
+value = 3
+```
+
+`Cps#and_then`'s block accepts the result of the previous operation and
+returns the `Cps` object representing the next operation to perform.
+
+As it turns out, the best way to factor an operation is as a
+method that
+
+- accepts the operation arguments, and
+- returns a `Cps` object
+
+as seen above.
+
+### Sequences
+
+As you can imagine, long `and_then` chains can get syntactially messy.
+This is where `seq` comes in.
+
+```
+def add1_and_print(n)
+  seq do |th|
+    th.en {     add1(n)        }
+    th.en { |v| print_value(v) }
+  end
+end
+```
+
+This is still not ideal; it would be nice to have a way to bind `v` to
+the result of `add1(n)` rather than introducing a parameter on the
+line below. `seql` contains some magic that allows us to do
+exactly this. (It also eliminates the weird `th.en`.)
+
+```ruby
+def add1_and_print(n)
+  seql do
+    let(:v)  { add1(n)        }
+    and_then { print_value(v) }
+  end
+end
+```
+
+Another benefit of `seql` and `let` is that it makes `v` visible to
+_all_ subsequent `and_then` blocks, not just the immediately following
+one.
+
+But what if we want to introduce a non-CPS operation into our
+sequence?
+
+```ruby
+def add1_and_print(n)
+  seql do
+    let(:v)  { add1(n)        }
+    and_then { puts 'added!'  }
+    and_then { print_value(v) }
+  end
+end
+```
+```
+2.2.2 :061 > add1_and_print(2).run
+added!
+RuntimeError: 'and_then' block did not return a Cps object.
+```
+
+This fails because it violates the requirement that the `and_then`
+block return a `Cps` object, and `puts` does not. `lift` returns a
+`Cps` object for a block of direct-style code.
+
+```ruby
+def add1_and_print(n)
+  seql do
+    let(:v)  { add1(n)                }
+    and_then { lift { puts 'added!' } }
+    and_then { print_value(v)         }
+  end
+end
+```
+```
+2.2.2 :053 > add1_and_print(2).run
+added!
+value = 3
+```
+
+Finally, a little clean up:
+
+```ruby
+def add1_and_print(n)
+  seql do
+    let(:v)  { add1(n)        }
+    and_lift { puts 'added!'  }
+    and_then { print_value(v) }
+  end
+end
+```
+
+### `Mercury::Monadic`
+
+`Mercury::Monadic` simply wraps `Mercury` so that the methods return
+`Cps` objects rather than accepting an explicit continuation.
+
+```ruby
+seql do
+  let(:m)  { Mercury::Monadic.open    }
+  and_then { m.start_listener(source) }
+  ...
+end
+```
+
+It is particularly useful when writing tests.
+
+
+Design Details
+--------------
+
+#### `Mercury#republish`
+
+This method publishes a copy of the message to the _back_ of the
+queue, then acks the original message. This is a similar operation to
+ack/nack/reject. It is only applicable to messages received by
+workers, since listener messages cannot be acknowledged. Unlike other
+acknowledgements, `#republish` takes a continuation (due to the
+publish operation), so the method is located on `Mercury` rather than
+`Mercury::ReceivedMessage`.
+
+It is important that the message is republished to the AMQP _queue_
+and not the _source_. Acknowledgement is a worker concern. If two
+different worker pools were working off a common source, it wouldn't
+make sense for pool A to get a duplicate message because pool B failed
+to handle the message.
+
+AMQP allows publishing to a particular queue by sending the message to
+the [default exchange][default_exchange], specifying the queue name as
+the routing key. Thus, if the message was originally sent with a
+non-empty routing key, that information is lost. Some clients rely on
+the routing key/tag to dictate behavior (by reading
+`Mercury::ReceivedMessage#tag`). To avoid breaking such clients,
+republish propagates the original tag in a header and reports this value
+as the tag on the republished message.
+
+Republishing also introduces a `Republish-Count` header and
+corresponding attribute `Mercury::ReceivedMessage#republish_count`.
+This value is incremented each time the message is republished.
+Clients may want to check this value and act differently if it exceeds
+some threshold.
+
+
 Design Decisions
 ----------------
 
@@ -216,3 +426,4 @@ is being intentionally ignored.
 [logatron]: https://github.com/indigobio/logatron
 [em_defer]: http://www.rubydoc.info/github/eventmachine/eventmachine/EventMachine.defer
 [fiber_defer]: https://github.com/indigobio/abstractivator/blob/master/lib/abstractivator/fiber_defer.rb
+[default_exchange]: https://www.rabbitmq.com/tutorials/amqp-concepts.html

data/lib/mercury/fake/queued_message.rb

@@ -9,7 +9,7 @@ class Mercury
 
       def initialize(queue, msg, tag, headers, is_ackable)
         metadata = Metadata.new(tag, headers, proc{queue.ack_or_reject_message(self)}, proc{queue.nack(self)})
-        @received_msg = ReceivedMessage.new(msg, metadata, is_ackable: is_ackable)
+        @received_msg = ReceivedMessage.new(msg, metadata, work_queue_name: is_ackable ? queue.worker : nil)
         @headers = headers
         @delivered = false
       end

data/lib/mercury/fake.rb

@@ -1,5 +1,7 @@
 require 'securerandom'
 require 'delegate'
+require 'active_support/core_ext/hash/keys'
+require 'active_support/core_ext/object/deep_dup'
 require 'mercury/received_message'
 require 'mercury/fake/domain'
 require 'mercury/fake/metadata'
@@ -43,7 +45,15 @@ class Mercury
 
     def publish(source_name, msg, tag: '', headers: {}, &k)
       guard_public(k)
-      queues.values.select{|q| q.binds?(source_name, tag)}.each{|q| q.enqueue(roundtrip(msg), tag, headers)}
+      queues.values.select{|q| q.binds?(source_name, tag)}.each{|q| q.enqueue(roundtrip(msg), tag, headers.stringify_keys)}
+      ret(k)
+    end
+
+    def republish(msg, &k)
+      guard_public(k)
+      msg.ack
+      queue = queues.values.detect{|q| q.worker == msg.work_queue_name}
+      queue.enqueue(roundtrip(msg.content), msg.tag, Mercury.increment_republish_count(msg))
       ret(k)
     end
 
@@ -58,7 +68,7 @@ class Mercury
     def start_worker_or_listener(source_name, handler, tag_filter, worker_group=nil, &k)
       guard_public(k)
       tag_filter ||= '#'
-      q = ensure_queue(source_name, tag_filter, !!worker_group, worker_group)
+      q = ensure_queue(source_name, tag_filter, worker_group)
       ret(k) # it's important we show the "start" operation finishing before delivery starts (in add_subscriber)
       q.add_subscriber(Subscriber.new(handler, @parallelism))
     end
@@ -102,7 +112,8 @@ class Mercury
       ws.read(ws.write(msg))
     end
 
-    def ensure_queue(source, tag_filter, require_ack, worker=nil)
+    def ensure_queue(source, tag_filter, worker)
+      require_ack = worker != nil
       worker ||= SecureRandom.uuid
       queues.fetch(unique_queue_name(source, tag_filter, worker)) do |k|
         queues[k] = Queue.new(source, tag_filter, worker, require_ack)

data/lib/mercury/mercury.rb

@@ -5,6 +5,9 @@ require 'mercury/received_message'
 require 'logatron/logatron'
 
 class Mercury
+  ORIGINAL_TAG_HEADER = 'Original-Tag'.freeze
+  REPUBLISH_COUNT_HEADER = 'Republish-Count'.freeze
+
   attr_reader :amqp, :channel, :logger
 
   def self.open(logger: Logatron, **kws, &k)
@@ -60,17 +63,34 @@ class Mercury
     # The amqp gem caches exchange objects, so it's fine to
     # redeclare the exchange every time we publish.
     with_source(source_name) do |exchange|
-      payload = write(msg)
-      pub_opts = Mercury.publish_opts(tag, headers)
-      if publisher_confirms_enabled
-        expect_publisher_confirm(k)
-        exchange.publish(payload, **pub_opts)
-      else
-        exchange.publish(payload, **pub_opts, &k)
-      end
+      publish_internal(exchange, msg, tag, headers, &k)
+    end
+  end
+
+  # Places a copy of the message at the back of the queue, then acks
+  # the original message.
+  def republish(msg, &k)
+    guard_public(k)
+    raise 'Only messages from a work queue can be republished' unless msg.work_queue_name
+    headers = Mercury.increment_republish_count(msg).merge(ORIGINAL_TAG_HEADER => msg.tag)
+    publish_internal(@channel.default_exchange, msg.content, msg.work_queue_name, headers) do
+      msg.ack
+      k.call
     end
   end
 
+  def publish_internal(exchange, msg, tag, headers, &k)
+    payload = write(msg)
+    pub_opts = Mercury.publish_opts(tag, headers)
+    if publisher_confirms_enabled
+      expect_publisher_confirm(k)
+      exchange.publish(payload, **pub_opts)
+    else
+      exchange.publish(payload, **pub_opts, &k)
+    end
+  end
+  private :publish_internal
+
   def self.publish_opts(tag, headers)
     { routing_key: tag, persistent: true, headers: Logatron.http_headers.merge(headers) }
   end
@@ -80,7 +100,7 @@ class Mercury
     with_source(source_name) do |exchange|
       with_listener_queue(exchange, tag_filter) do |queue|
         queue.subscribe(ack: false) do |metadata, payload|
-          handler.call(make_received_message(payload, metadata, false))
+          handler.call(make_received_message(payload, metadata))
         end
         k.call
       end
@@ -92,7 +112,7 @@ class Mercury
     with_source(source_name) do |exchange|
       with_work_queue(worker_group, exchange, tag_filter) do |queue|
         queue.subscribe(ack: true) do |metadata, payload|
-          handler.call(make_received_message(payload, metadata, true))
+          handler.call(make_received_message(payload, metadata, work_queue_name: worker_group))
         end
         k.call
       end
@@ -137,6 +157,8 @@ class Mercury
 
   private
 
+  LOGATRAON_MSG_ID_HEADER = 'X-Ascent-Log-Id'.freeze
+
   # In AMQP, queue consumers ack requests after handling them. Unacked messages
   # are automatically returned to the queue, guaranteeing they are eventually handled.
   # Services often ack one request while publishing related messages. Ideally, these
@@ -189,9 +211,9 @@ class Mercury
     end
   end
 
-  def make_received_message(payload, metadata, is_ackable)
-    msg = ReceivedMessage.new(read(payload), metadata, is_ackable: is_ackable)
-    Logatron.msg_id = msg.headers['X-Ascent-Log-Id']
+  def make_received_message(payload, metadata, work_queue_name: nil)
+    msg = ReceivedMessage.new(read(payload), metadata, work_queue_name: work_queue_name)
+    Logatron.msg_id = msg.headers[LOGATRAON_MSG_ID_HEADER]
     msg
   end
 
@@ -295,6 +317,12 @@ class Mercury
     end
   end
 
+  # @param msg [Mercury::ReceivedMessage]
+  # @return [Hash] the headers with republish count incremented
+  def self.increment_republish_count(msg)
+    msg.headers.merge(REPUBLISH_COUNT_HEADER => msg.republish_count + 1)
+  end
+
   def guard_public(k, initializing: false)
     Mercury.guard_public(@amqp.nil?, k, initializing: initializing)
   end

data/lib/mercury/monadic.rb

@@ -25,6 +25,7 @@ class Mercury
     end
 
     wrap(:publish)
+    wrap(:republish)
     wrap(:start_listener)
     wrap(:start_worker)
     wrap(:delete_source)

data/lib/mercury/received_message.rb

@@ -1,19 +1,23 @@
 class Mercury
   class ReceivedMessage
-    attr_reader :content, :metadata, :action_taken
+    attr_reader :content, :metadata, :action_taken, :work_queue_name
 
-    def initialize(content, metadata, is_ackable: false)
+    def initialize(content, metadata, work_queue_name: nil)
       @content = content
       @metadata = metadata
-      @is_ackable = is_ackable
+      @work_queue_name = work_queue_name
     end
 
     def tag
-      metadata.routing_key
+      headers[Mercury::ORIGINAL_TAG_HEADER] || metadata.routing_key
     end
 
     def headers
-      metadata.headers || {}
+      (metadata.headers || {}).dup
+    end
+
+    def republish_count
+      (metadata.headers[Mercury::REPUBLISH_COUNT_HEADER] || 0).to_i
     end
 
     def ack
@@ -33,8 +37,12 @@ class Mercury
 
     private
 
+    def is_ackable
+      @work_queue_name != nil
+    end
+
     def performing_action(action)
-      @is_ackable or raise "This message is not #{action}able"
+      is_ackable or raise "This message is not #{action}able"
       if @action_taken
         raise "This message was already #{@action_taken}ed"
       end

data/lib/mercury/version.rb

@@ -1,3 +1,3 @@
 class Mercury
-  VERSION = '0.4.0'
+  VERSION = '0.5.0'
 end

data/mercury_amqp.gemspec

@@ -31,4 +31,5 @@ Gem::Specification.new do |spec|
   spec.add_runtime_dependency 'bunny', '~> 2.1'
   spec.add_runtime_dependency 'binding_of_caller', '~> 0.7'
   spec.add_runtime_dependency 'logatron', '~> 0'
+  spec.add_runtime_dependency 'activesupport', '~> 4.0'
 end

data/spec/lib/mercury/monadic_spec.rb

@@ -121,6 +121,34 @@ describe Mercury::Monadic do
     end
   end
 
+  itt 'republishes' do
+    test_with_mercury do |m|
+      msgs = []
+      seql do
+        and_then { m.start_worker(worker, source, &msgs.method(:push)) }
+        and_then { m.publish(source, msg, tag: 'foo', headers: {bar: 123}) }
+        and_then { wait_until { msgs.size == 1 } }
+        and_lift do
+          expect(msgs.last.tag).to eql 'foo'
+          expect(msgs.last.headers['bar']).to eql 123
+          expect(msgs.last.republish_count).to eql 0
+        end
+        and_then { m.republish(msgs.last) }
+        and_then { wait_until { msgs.size == 2 } }
+        and_lift do
+          expect(msgs.last.tag).to eql 'foo'          # preserves the tag
+          expect(msgs.last.headers['bar']).to eql 123 # preserves the headers
+          expect(msgs.last.republish_count).to eql 1  # increments the republish count
+        end
+        and_then { m.republish(msgs.last) }
+        and_then { wait_until { msgs.size == 3 } }
+        and_lift do
+          expect(msgs.last.republish_count).to eql 2  # can republish a republished message
+        end
+      end
+    end
+  end
+
   it 'propagates logatron headers' do
     real_msg_id = SecureRandom.uuid
     Logatron.msg_id = real_msg_id

data/spec/lib/mercury/received_message_spec.rb

@@ -56,11 +56,11 @@ describe Mercury::ReceivedMessage do
   end
 
   def make_actionable
-    Mercury::ReceivedMessage.new('hello', make_metadata, is_ackable: true)
+    Mercury::ReceivedMessage.new('hello', make_metadata, work_queue_name: 'foo')
   end
 
   def make_non_actionable
-    Mercury::ReceivedMessage.new('hello', make_metadata, is_ackable: false)
+    Mercury::ReceivedMessage.new('hello', make_metadata, work_queue_name: nil)
   end
 
   def make_metadata

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: mercury_amqp
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.5.0
 platform: ruby
 authors:
 - Peter Winton
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-04-28 00:00:00.000000000 Z
+date: 2016-05-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -178,6 +178,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: activesupport
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.0'
 description: Abstracts common patterns used with AMQP
 email:
 - wintonpc@gmail.com