circuitry 2.0.0 → 2.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 7d3f37801853268fc7adb65ff0f4d4c763f0f098
-  data.tar.gz: aae27a8ce4d592ff7e817c0aee62e2c82a973026
+  metadata.gz: 14368085a767ad616196eb22569c3c981af716d0
+  data.tar.gz: 639f43f5b3e192241e5b142c5251331ee8f86362
 SHA512:
-  metadata.gz: bdd6ebe7c82eec7d98dafbfad50cd056c554db06a9fbd4f41d749c3ed94bfa18efafbd693404ca3863804e007fa941807acee55376db52dbba2e0abb99457d5a
-  data.tar.gz: 01b6255d1c8d3826650cb4d7a263b71c28d310e7a69fb207b1e322495741531e36b21bd5d263f5702665a328077b6c7a5d19f3a4d6ca872d3162bf170d3f5ed7
+  metadata.gz: 15d2f7452c6f1324850b6b83a825ecc9c5bc7895319660b785917625dbf0a991f92ce7f4303c239df4eb520b8c128e427fd69c85069e687986fdc198d693266b
+  data.tar.gz: 8b4073842d2dca9813ee809ffcbab89ce8949ba148edbfa7d32531be5a79764eccc4f067489decbccccd143f86afa7db8767c55b34b9be64337ca9bc18aa8b41

data/CHANGELOG.md

@@ -1,9 +1,13 @@
-## Circuitry 2.0.0 (Jan 27, 2016)
+## Circuitry 2.1.0 (Jan 28, 2016)
 
-* Added subscriber_queue_name config *Brandon Croft*
-* Added publisher_topic_names config *Brandon Croft*
-* Added CLI and rake provisioning of queues and topics as defined by config *Brandon Croft*
-* Removed the requirement to provide a SQS URL to the subscriber *Brandon Croft*
+* Added publisher and subscriber middleware. *Matt Huggins*
+
+## Circuitry 2.0.0 (Jan 28, 2016)
+
+* Added subscriber_queue_name config. *Brandon Croft*
+* Added publisher_topic_names config. *Brandon Croft*
+* Added CLI and rake provisioning of queues and topics as defined by config. *Brandon Croft*
+* Removed the requirement to provide a SQS URL to the subscriber. *Brandon Croft*
 
 ## Circuitry 1.4.1 (Jan 21, 2016)
 

data/README.md

@@ -87,18 +87,26 @@ Available configuration options include:
   managing shared resources such as database connections that require closing,
   It is only called when implementing the `:fork` async strategy. *(optional,
   default: `nil`)*
+* `publisher_topic_names`: An array of topic names that your publishing application will
+  publish on. This configuration is only used during provisioning via `rake circuitry:setup`
 * `subscriber_queue_name`: The name of the SQS queue that your subscriber application
   will listen to. This queue will be created or configured during `rake circuitry:setup`
   *(optional, default: `nil`)*
 * `subscriber_dead_letter_queue_name`: The name of the SQS dead letter queue that will be
   used after all retries fail. This queue will be created and configured during `rake
   circuitry:setup` *(optional, default: `<subscriber_queue_name>-failures`)*
-* `publisher_topic_names`: An array of topic names that your publishing application will
-  publish on. This configuration is only used during provisioning via `rake circuitry:setup`
+* `publisher_middleware`: A chain of middleware that sent messages must go through.
+  Please refer to the [Middleware](#middleware) section for more details regarding this
+  option.
+* `subscriber_middleware`: A chain of middleware that received messages must go through.
+  Please refer to the [Middleware](#middleware) section for more details regarding this
+  option.
 
 ### Provisioning
 
-You can automatically provision SQS queues, SNS topics, and the subscriptions between them using two methods: the circuitry CLI or the `rake circuitry:setup` task. The rake task will provision the subscriber queue and publishing topics that are configured within your application.
+You can automatically provision SQS queues, SNS topics, and the subscriptions between them using
+two methods: the circuitry CLI or the `rake circuitry:setup` task. The rake task will provision the
+subscriber queue and publishing topics that are configured within your application.
 
 ```ruby
 Circuitry.config do |c|
@@ -107,7 +115,9 @@ Circuitry.config do |c|
 end
 ```
 
-When provisioning, a dead letter queue is also created using the name "<queue_name>-failures" and a redrive policy of 8 retries to that dead letter queue is configured. You can customize the dead letter queue name in your configuration.
+When provisioning, a dead letter queue is also created using the name "<queue_name>-failures" and a
+redrive policy of 8 retries to that dead letter queue is configured. You can customize the dead
+letter queue name in your configuration.
 
 Run `ruby bin/circuitry help provision` for help using CLI provisioning.
 
@@ -152,8 +162,8 @@ publisher.publish('my-topic-name', obj)
 
 ### Subscribing
 
-Subscribing is done via the `Circuitry.subscribe` method. It accepts a block for processing each message. This method **indefinitely
-blocks**, processing messages as they are enqueued.
+Subscribing is done via the `Circuitry.subscribe` method. It accepts a block for processing each
+message. This method **indefinitely blocks**, processing messages as they are enqueued.
 
 ```ruby
 Circuitry.subscribe do |message, topic_name|
@@ -422,6 +432,80 @@ connection = PG.connect(...)
 Circuitry.config.lock_strategy = DatabaseLockStrategy.new(connection: connection)
 ```
 
+### Middleware
+
+Circuitry middleware can be used to perform additional processing around a message
+being sent by a publisher or received by a subscriber.  Some examples of processing
+that belong here are monitoring or encryption specific to your application.
+
+Middleware can be added to the publisher, the subscriber, or both.  A middleware
+class is defined by an (optional) `#initialize` method that accepts any number of
+arguments, as well as a `#call` method that accepts the `topic` string, `message`
+string, and a block for continuing processing.
+
+For example, a simple logging middleware might look something like the following:
+
+```ruby
+class LoggerMiddleware
+  attr_reader :namespace, :logger
+
+  def initialize(namespace:, logger: Logger.new(STDOUT))
+    self.namespace = namespace
+    self.logger = logger
+  end
+  
+  def call(topic, message)
+    logger.info("#{namespace} (start): #{topic} - #{message}")
+    yield
+  ensure
+    logger.info("#{namespace} (done): #{topic} - #{message}")
+  end
+
+  private
+
+  attr_writer :namespace, :logger
+end
+```
+
+Adding the middleware to the stack happens through the Circuitry config.
+
+```ruby
+Circuitry.config do |config|
+  # single-line format
+  circuitry.publisher_middleware.add LoggerMiddleware, namespace: 'publisher'
+  circuitry.subscriber_middleware.add LoggerMiddleware, namespace: 'subscriber', logger: Rails.logger
+
+  # block format
+  circuitry.publisher_middleware do |chain|
+    chain.add LoggerMiddleware, namespace: 'publisher'
+  end
+
+  circuitry.subscriber_middleware do |chain|
+    chain.add LoggerMiddleware, namespace: 'subscriber', logger: Rails.logger
+  end
+end
+```
+
+Both `publisher_middleware` and `subscriber_middleware` respond to a handful of methods that can be
+used for configuring your middleware:
+
+* `#add`: Appends a middleware class to the end of the chain.  If the class already exists, it is
+  replaced.
+  * `middleware.add NewMiddleware, arg1, arg2, ...`
+* `#prepend`: Prepends a middleware class to the beginning of the chain.  If the class already
+  exists, it is replaced.
+  * `middleware.prepend NewMiddleware, arg1, arg2, ...`
+* `#remove`: Removes a middleware class from anywhere in the chain.
+  * `middleware.remove NewMiddleware`
+* `#insert_before`: Injects a middleware class before another middleware class in the chain.  If
+  the other class does not exist in the chain, this behaves the same as `#prepend`.
+  * `middleware.insert_before ExistingMiddleware, NewMiddleware, arg1, arg2...`
+* `#insert_after`: Injects a middleware class after another middleware class in the chain.  If the
+  other class does not exist in the chain, this behaves the same as `#add`.
+  * `middleware.insert_after ExistingMiddleware, NewMiddleware, arg1, arg2...`
+* `#clear`: Removes all middleware classes from the chain.
+  * `middleware.clear`
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run

data/lib/circuitry.rb

@@ -5,6 +5,7 @@ require 'circuitry/locks/memcache'
 require 'circuitry/locks/memory'
 require 'circuitry/locks/noop'
 require 'circuitry/locks/redis'
+require 'circuitry/middleware/chain'
 require 'circuitry/processor'
 require 'circuitry/processors/batcher'
 require 'circuitry/processors/forker'

data/lib/circuitry/configuration.rb

@@ -34,6 +34,18 @@ module Circuitry
       super || "#{subscriber_queue_name}-failures"
     end
 
+    def subscriber_middleware
+      @subscriber_middleware ||= Circuitry::Middleware::Chain.new
+      yield @subscriber_middleware if block_given?
+      @subscriber_middleware
+    end
+
+    def publisher_middleware
+      @publisher_middleware ||= Circuitry::Middleware::Chain.new
+      yield @publisher_middleware if block_given?
+      @publisher_middleware
+    end
+
     def aws_options
       {
         access_key_id:     access_key,

data/lib/circuitry/middleware/chain.rb

@@ -0,0 +1,82 @@
+require 'circuitry/middleware/entry'
+
+module Circuitry
+  module Middleware
+    class Chain
+      include Enumerable
+
+      def each(&block)
+        entries.each(&block)
+      end
+
+      def entries
+        @entries ||= []
+      end
+
+      def add(klass, *args)
+        remove(klass) if exists?(klass)
+        entries << Entry.new(klass, *args)
+      end
+
+      def remove(klass)
+        entries.delete_if { |entry| entry.klass == klass }
+      end
+
+      def prepend(klass, *args)
+        remove(klass) if exists?(klass)
+        entries.unshift(Entry.new(klass, *args))
+      end
+
+      def insert_before(old_klass, new_klass, *args)
+        new_entry = build_or_replace_entry(new_klass, *args)
+        i = entries.index { |entry| entry.klass == old_klass } || 0
+        entries.insert(i, new_entry)
+      end
+
+      def insert_after(old_klass, new_klass, *args)
+        new_entry = build_or_replace_entry(new_klass, *args)
+        i = entries.index { |entry| entry.klass == old_klass } || entries.size - 1
+        entries.insert(i + 1, new_entry)
+      end
+
+      def exists?(klass)
+        any? { |entry| entry.klass == klass }
+      end
+
+      def build
+        map(&:build)
+      end
+
+      def clear
+        entries.clear
+      end
+
+      def invoke(*args)
+        chain = build.dup
+
+        traverse_chain = -> do
+          if chain.empty?
+            yield
+          else
+            chain.shift.call(*args, &traverse_chain)
+          end
+        end
+
+        traverse_chain.call
+      end
+
+      private
+
+      def build_or_replace_entry(klass, *args)
+        i = entries.index { |entry| entry.klass == klass }
+        entry = i.nil? ? Entry.new(klass, *args) : entries.delete_at(i)
+
+        if entry.args == args
+          entry
+        else
+          Entry.new(klass, *args)
+        end
+      end
+    end
+  end
+end

data/lib/circuitry/middleware/entry.rb

@@ -0,0 +1,20 @@
+module Circuitry
+  module Middleware
+    class Entry
+      attr_reader :klass, :args
+
+      def initialize(klass, *args)
+        self.klass = klass
+        self.args  = args
+      end
+
+      def build
+        klass.new(*args)
+      end
+
+      private
+
+      attr_writer :klass, :args
+    end
+  end
+end

data/lib/circuitry/provisioner.rb

@@ -4,12 +4,11 @@ require 'circuitry/subscription_creator'
 
 module Circuitry
   class Provisioner
-    attr_reader :log
-    attr_reader :config
+    attr_reader :config, :logger
 
     def initialize(config, logger: Logger.new(STDOUT))
-      @config = config
-      @log = logger
+      self.config = config
+      self.logger = logger
     end
 
     def run
@@ -20,13 +19,15 @@ module Circuitry
 
     private
 
+    attr_writer :config, :logger
+
     def create_queue
       safe_aws('Create Queue') do
         queue = Circuitry::QueueCreator.find_or_create(
           config.subscriber_queue_name,
           dead_letter_queue_name: config.subscriber_dead_letter_queue_name
         )
-        log.info "Created queue #{queue.url}"
+        logger.info "Created queue #{queue.url}"
         queue
       end
     end
@@ -35,7 +36,7 @@ module Circuitry
       safe_aws('Create Topics') do
         config.publisher_topic_names.map do |topic_name|
           topic = Circuitry::TopicCreator.find_or_create(topic_name)
-          log.info "Created topic #{topic.name}"
+          logger.info "Created topic #{topic.name}"
           topic
         end
       end
@@ -44,7 +45,7 @@ module Circuitry
     def subscribe_topics(queue, topics)
       safe_aws('Subscribe Topics') do
         Circuitry::SubscriptionCreator.subscribe_all(queue, topics)
-        log.info "Subscribed all topics to #{queue.name}"
+        logger.info "Subscribed all topics to #{queue.name}"
         true
       end
     end
@@ -52,7 +53,7 @@ module Circuitry
     def safe_aws(desc)
       yield
     rescue Aws::SQS::Errors::AccessDenied
-      log.fatal("#{desc}: Access denied. Check your configured credentials.")
+      logger.fatal("#{desc}: Access denied. Check your configured credentials.")
       nil
     end
   end

data/lib/circuitry/publisher.rb

@@ -30,10 +30,12 @@ module Circuitry
       raise ArgumentError, 'object cannot be nil' if object.nil?
       raise PublishError, 'AWS configuration is not set' unless can_publish?
 
+      message = object.to_json
+
       if async?
-        process_asynchronously(& -> { publish_internal(topic_name, object) })
+        process_asynchronously { publish_internal(topic_name, message) }
       else
-        publish_internal(topic_name, object)
+        publish_internal(topic_name, message)
       end
     end
 
@@ -43,14 +45,16 @@ module Circuitry
 
     protected
 
-    def publish_internal(topic_name, object)
-      # TODO: Don't use ruby timeout.
-      # http://www.mikeperham.com/2015/05/08/timeout-rubys-most-dangerous-api/
-      Timeout.timeout(timeout) do
-        logger.info("Publishing message to #{topic_name}")
+    def publish_internal(topic_name, message)
+      middleware.invoke(topic_name, message) do
+        # TODO: Don't use ruby timeout.
+        # http://www.mikeperham.com/2015/05/08/timeout-rubys-most-dangerous-api/
+        Timeout.timeout(timeout) do
+          logger.info("Publishing message to #{topic_name}")
 
-        topic = TopicCreator.find_or_create(topic_name)
-        sns.publish(topic_arn: topic.arn, message: object.to_json)
+          topic = TopicCreator.find_or_create(topic_name)
+          sns.publish(topic_arn: topic.arn, message: message)
+        end
       end
     end
 
@@ -67,5 +71,9 @@ module Circuitry
         !value.nil? && !value.empty?
       end
     end
+
+    def middleware
+      Circuitry.config.publisher_middleware
+    end
   end
 end

data/lib/circuitry/subscriber.rb

@@ -119,7 +119,7 @@ module Circuitry
     end
 
     def process_messages_asynchronously(messages, &block)
-      messages.each { |message| process_asynchronously(& -> { process_message(message, &block) }) }
+      messages.each { |message| process_asynchronously { process_message(message, &block) } }
     end
 
     def process_messages_synchronously(messages, &block)
@@ -139,21 +139,25 @@ module Circuitry
 
     def handle_message(message, &block)
       handled = try_with_lock(message.id) do
-        begin
-          # TODO: Don't use ruby timeout.
-          # http://www.mikeperham.com/2015/05/08/timeout-rubys-most-dangerous-api/
-          Timeout.timeout(timeout) do
-            block.call(message.body, message.topic.name)
-          end
-        rescue => e
-          logger.error("Error handling message #{message.id}: #{e}")
-          raise e
+        middleware.invoke(message.topic.name, message.body) do
+          handle_message_with_timeout(message, &block)
         end
       end
 
       logger.info("Ignoring duplicate message #{message.id}") unless handled
     end
 
+    # TODO: Don't use ruby timeout.
+    # http://www.mikeperham.com/2015/05/08/timeout-rubys-most-dangerous-api/
+    def handle_message_with_timeout(message, &block)
+      Timeout.timeout(timeout) do
+        block.call(message.body, message.topic.name)
+      end
+    rescue => e
+      logger.error("Error handling message #{message.id}: #{e}")
+      raise e
+    end
+
     def try_with_lock(handle)
       if lock.soft_lock(handle)
         begin
@@ -188,5 +192,9 @@ module Circuitry
         !value.nil? && !value.empty?
       end
     end
+
+    def middleware
+      Circuitry.config.subscriber_middleware
+    end
   end
 end

data/lib/circuitry/version.rb

@@ -1,3 +1,3 @@
 module Circuitry
-  VERSION = '2.0.0'
+  VERSION = '2.1.0'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: circuitry
 version: !ruby/object:Gem::Version
-  version: 2.0.0
+  version: 2.1.0
 platform: ruby
 authors:
 - Matt Huggins
@@ -254,6 +254,8 @@ files:
 - lib/circuitry/locks/noop.rb
 - lib/circuitry/locks/redis.rb
 - lib/circuitry/message.rb
+- lib/circuitry/middleware/chain.rb
+- lib/circuitry/middleware/entry.rb
 - lib/circuitry/processor.rb
 - lib/circuitry/processors/batcher.rb
 - lib/circuitry/processors/forker.rb
@@ -291,7 +293,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.5.1
+rubygems_version: 2.4.8
 signing_key: 
 specification_version: 4
 summary: Decouple ruby applications using Amazon SNS fanout with SQS processing.