circuitry 1.0.0 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 2399e4bc4f1e1d82e117415684b20c2920e34020
-  data.tar.gz: 9b497dde60c50e60b945ef80be7dbe07382755e8
+  metadata.gz: e63a7c32a7c197c43ef244bdb5b3be5ae1544f39
+  data.tar.gz: f99bee8e6cc734e0ae334d31de01eb70851717e3
 SHA512:
-  metadata.gz: b026a20b13d88fcfa09b7ab069578c236fca5dff3aa94f80b98e13aa4d2520b195a4c0cbed62f679211ee27cc54bb05be2f8657f6b738e63620422883b692be7
-  data.tar.gz: 6248ef0005b4c98c9aa3d50cdf977a3426b78d8f8eac8a420ed4066ae258b2128e614e1ef091e5b829ba7eea1610546ec7b27692f554e2e2572f1fe5777149c8
+  metadata.gz: 4bb4da24b7d24c81b212c3c8f750d8bed4c85e8bde06f96e25f08c5d6291e1bed908948eec337caa193333e821cd02a01cfc94d5c86dcb3eeeddd24fab893da5
+  data.tar.gz: a54b2dc0d13eb5172e869163bef080e2a3095f09e3fa832ce6b79d0a1b508ec9fdf03610c4a98be83a3c7127b914ea82a398a935beb05d12f2e5891a3ed00933

data/README.md

@@ -36,6 +36,8 @@ Circuitry.config do |c|
     HoneyBadger.notify(error)
     HoneyBadger.flush
   end
+  c.publish_async_strategy = :batch
+  c.subscribe_async_strategy = :thread
 end
 ```
 
@@ -52,12 +54,27 @@ Available configuration options include:
 * `error_handler`: An object that responds to `call` with two arguments: the
   deserialized message contents and the topic name used when publishing to SNS.
   *(optional, default: `nil`)*
+* `publish_async_strategy`: One of `:fork`, `:thread`, or `:batch` that
+  determines how asynchronous publish requests are processed.  *(optional,
+  default: `:fork`)*
+  * `:fork`: Forks a detached child process that immediately sends the request.
+  * `:thread`: Creates a new thread that immediately sends the request.  Because
+    threads are not guaranteed to complete when the process exits, completion can
+    be ensured by calling `Circuitry.flush`.
+  * `:batch`: Stores the request in memory to be submitted later.  Batched
+    requests must be manually sent by calling `Circuitry.flush`.
+* `subscribe_async_strategy`: One of `:fork` or `:thread` that determines how
+  asynchronous subscribe requests are processed.  *(optional, default: `:fork`)*
+  * `:fork`: Forks a detached child process that immediately begins querying the
+    queue.
+  * `:thread`: Creates a new thread that immediately sends begins querying the
+    queue.
 
 ### Publishing
 
 Publishing is done via the `Circuitry.publish` method.  It accepts a topic name
-the represents the SNS topic along with any non-nil object, representing the data
-to be serialized.  Whatever object is called will have its `to_json` method
+that represents the SNS topic along with any non-nil object, representing the
+data to be serialized.  Whatever object is called will have its `to_json` method
 called for serialization.
 
 ```ruby
@@ -68,9 +85,11 @@ Circuitry.publish('any-topic-name', obj)
 The `publish` method also accepts options that impact instantiation of the
 `Publisher` object, which currently includes the following options.
 
-* `:async` - Whether or not publishing should occur in the background.  Please
-   refer to the [Asynchronous Support](#asynchronous-support) section for more
-   details regarding this option.  (default: false)
+* `:async` - Whether or not publishing should occur in the background.  Accepts
+  one of `:fork`, `:thread`, `:batch`, `true`, or `false`.  Passing `true` uses
+  the `publish_async_strategy` value from the gem configuration.  Please refer to
+  the [Asynchronous Support](#asynchronous-support) section for more details
+  regarding this option.  *(default: `false`)*
 
 ```ruby
 obj = { foo: 'foo', bar: 'bar' }
@@ -89,8 +108,8 @@ publisher.publish('my-topic-name', obj)
 ### Subscribing
 
 Subscribing is done via the `Circuitry.subscribe` method.  It accepts an SQS queue
-URL and takes a block for processing each message.  This method performs
-synchronously by default, and as such does not return.
+URL and takes a block for processing each message.  This method **indefinitely
+blocks**, processing messages as they are enqueued.
 
 ```ruby
 Circuitry.subscribe('https://sqs.REGION.amazonaws.com/ACCOUNT-ID/QUEUE-NAME') do |message, topic_name|
@@ -101,14 +120,18 @@ end
 The `subscribe` method also accepts options that impact instantiation of the
 `Subscriber` object, which currently includes the following options.
 
-* `:async` - Whether or not subscribing should occur in the background.  Please
-   refer to the [Asynchronous Support](#asynchronous-support) section for more
-   details regarding this option.  (default: false)
+* `:async` - Whether or not subscribing should occur in the background.  Accepts
+  one of `:fork`, `:thread`, `true`, or `false`.  Passing `true` uses the
+  `subscribe_async_strategy` value from the gem configuration.  Passing an
+  asynchronous value will cause messages to be handled concurrently, meaning
+  that queued messages *might not be processed in the order they're received*.
+  Please refer to the [Asynchronous Support](#asynchronous-support) section for
+  more details regarding this option.  *(default: `false`)*
 * `:wait_time` - The number of seconds to wait for messages while connected to
   SQS.  Anything above 0 results in long-polling, while 0 results in
-  short-polling.  (default: 10)
+  short-polling.  *(default: 10)*
 * `:batch_size` - The number of messages to retrieve in a single SQS request.
-  (default: 10)
+  *(default: 10)*
 
 ```ruby
 Circuitry.subscribe('https://...', async: true, wait_time: 60, batch_size: 20) do |message, topic_name|
@@ -129,17 +152,22 @@ end
 
 ### Asynchronous Support
 
-Publishing or subscribing asynchronously occurs by forking a child process.  That
-child is then detached so that your application does not need to worry about
-waiting for the process to finish.
+Publishing supports three asynchronous strategies (forking, threading, and
+batching) while subscribing supports two (forking and threading).
+
+#### Forking
+
+When forking a child process, that child is detached so that your application
+does not need to worry about waiting for the process to finish.  Forked requests
+begin processing immediately and do not have any overhead in terms of waiting for
+them to complete.
 
 There are two important notes regarding forking in general as it relates to
 asynchronous support:
 
 1. Forking is not supported on all platforms (e.g.: Windows and NetBSD 4),
-   requiring that your implementation use synchronous requests in such
-   circumstances.  You can determine if asynchronous requests will work by
-   calling `Circuitry.platform_supports_async?`.
+   requiring that your implementation use synchronous requests or an alternative
+   asynchronous strategy in such circumstances.
 
 2. Forking results in resources being copied from the parent process to the child
    process.  In order to prevent database connection errors and the like, you
@@ -164,6 +192,18 @@ asynchronous support:
    Refer to your adapter's documentation to determine how resources are handled
    with regards to forking.
 
+#### Threading
+
+Threaded publish and subscribe requests begin processing immediately.  Unlike
+forking, it's up to you to ensure that all threads complete before your
+application exits.  This can be done by calling `Circuitry.flush`.
+
+#### Batching
+
+Batched publish and subscribe requests are queued in memory and do not begin
+processing until you explicit flush them.  This can be done by calling
+`Circuitry.flush`.
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run

data/circle.yml

@@ -1,3 +1,7 @@
+machine:
+  ruby:
+    version: 2.2.0
+
 dependencies:
   pre:
     - gem install bundler -v 1.8.9

data/lib/circuitry/concerns/async.rb

@@ -1,17 +1,67 @@
+require 'circuitry/processors/batcher'
+require 'circuitry/processors/forker'
+require 'circuitry/processors/threader'
+
 module Circuitry
   class NotSupportedError < StandardError; end
 
   module Concerns
     module Async
+      attr_reader :async
+
+      def self.included(base)
+        base.extend(ClassMethods)
+      end
+
+      module ClassMethods
+        def async_strategies
+          [:fork, :thread, :batch]
+        end
+
+        def default_async_strategy
+          raise NotImplementedError, "#{name} must implement class method `default_async_strategy`"
+        end
+      end
+
       def process_asynchronously(&block)
-        raise NotSupportedError, 'Your platform does not support forking' unless platform_supports_async?
+        send(:"process_via_#{async}", &block)
+      end
+
+      def async=(value)
+        value = case value
+          when false, nil then false
+          when true then self.class.default_async_strategy
+          when *self.class.async_strategies then value
+          else raise ArgumentError, "Invalid value `#{value.inspect}`, must be one of #{[true, false].concat(self.class.async_strategies).inspect}"
+        end
+
+        if value == :fork && !platform_supports_forking?
+          raise NotSupportedError, 'Your platform does not support forking'
+        end
+
+        @async = value
+      end
+
+      def async?
+        !!async
+      end
+
+      private
+
+      def platform_supports_forking?
+        Process.respond_to?(:fork)
+      end
+
+      def process_via_fork(&block)
+        Processors::Forker.process(&block)
+      end
 
-        pid = fork(&block)
-        Process.detach(pid)
+      def process_via_thread(&block)
+        Processors::Threader.process(&block)
       end
 
-      def platform_supports_async?
-        Circuitry.platform_supports_async?
+      def process_via_batch(&block)
+        Processors::Batcher.process(&block)
       end
     end
   end

data/lib/circuitry/configuration.rb

@@ -10,6 +10,18 @@ module Circuitry
     attribute :region, String, default: 'us-east-1'
     attribute :logger, Logger, default: Logger.new(STDERR)
     attribute :error_handler
+    attribute :publish_async_strategy, Symbol, default: ->(page, attribute) { :fork }
+    attribute :subscribe_async_strategy, Symbol, default: ->(page, attribute) { :fork }
+
+    def publish_async_strategy=(value)
+      validate(value, Publisher.async_strategies)
+      super
+    end
+
+    def subscribe_async_strategy=(value)
+      validate(value, Subscriber.async_strategies)
+      super
+    end
 
     def aws_options
       {
@@ -18,5 +30,13 @@ module Circuitry
           region:                region,
       }
     end
+
+    private
+
+    def validate(value, permitted_values)
+      unless permitted_values.include?(value)
+        raise ArgumentError, "invalid value `#{value}`, must be one of #{permitted_values.inspect}"
+      end
+    end
   end
 end

data/lib/circuitry/processor.rb

@@ -0,0 +1,36 @@
+module Circuitry
+  module Processor
+    def process
+      raise NotImplementedError, "#{self} must implement class method `process`"
+    end
+
+    def flush
+      raise NotImplementedError, "#{self} must implement class method `flush`"
+    end
+
+    protected
+
+    def safely_process(&block)
+      begin
+        block.call
+      rescue => e
+        logger.error("Error handling message: #{e}")
+        error_handler.call(e) if error_handler
+      end
+    end
+
+    def pool
+      @pool ||= []
+    end
+
+    private
+
+    def logger
+      Circuitry.config.logger
+    end
+
+    def error_handler
+      Circuitry.config.error_handler
+    end
+  end
+end

data/lib/circuitry/processors/batcher.rb

@@ -0,0 +1,22 @@
+require 'circuitry/processor'
+
+module Circuitry
+  module Processors
+    module Batcher
+      class << self
+        include Processor
+
+        def process(&block)
+          raise ArgumentError, 'no block given' unless block_given?
+          pool << block
+        end
+
+        def flush
+          pool.each { |block| safely_process(&block) }
+        ensure
+          pool.clear
+        end
+      end
+    end
+  end
+end

data/lib/circuitry/processors/forker.rb

@@ -0,0 +1,19 @@
+require 'circuitry/processor'
+
+module Circuitry
+  module Processors
+    module Forker
+      class << self
+        include Processor
+
+        def process(&block)
+          pid = fork { safely_process(&block) }
+          Process.detach(pid)
+        end
+
+        def flush
+        end
+      end
+    end
+  end
+end

data/lib/circuitry/processors/threader.rb

@@ -0,0 +1,22 @@
+require 'circuitry/processor'
+
+module Circuitry
+  module Processors
+    module Threader
+      class << self
+        include Processor
+
+        def process(&block)
+          raise ArgumentError, 'no block given' unless block_given?
+          pool << Thread.new { safely_process(&block) }
+        end
+
+        def flush
+          pool.each(&:join)
+        ensure
+          pool.clear
+        end
+      end
+    end
+  end
+end

data/lib/circuitry/publisher.rb

@@ -17,7 +17,7 @@ module Circuitry
     def initialize(options = {})
       options = DEFAULT_OPTIONS.merge(options)
 
-      @async = !!options[:async]
+      self.async = options[:async]
     end
 
     def publish(topic_name, object)
@@ -41,8 +41,8 @@ module Circuitry
       end
     end
 
-    def async?
-      @async
+    def self.default_async_strategy
+      Circuitry.config.publish_async_strategy
     end
 
     private

data/lib/circuitry/subscriber.rb

@@ -26,10 +26,10 @@ module Circuitry
 
       options = DEFAULT_OPTIONS.merge(options)
 
-      @queue = queue
-      @async = !!options[:async]
-      @wait_time = options[:wait_time]
-      @batch_size = options[:batch_size]
+      self.queue = queue
+      self.async = options[:async]
+      self.wait_time = options[:wait_time]
+      self.batch_size = options[:batch_size]
     end
 
     def subscribe(&block)
@@ -40,28 +40,28 @@ module Circuitry
         return
       end
 
-      process = -> do
-        loop do
-          begin
-            receive_messages(&block)
-          rescue *CONNECTION_ERRORS => e
-            logger.error "Connection error to #{queue}: #{e}"
-            raise SubscribeError.new(e)
-          end
+      loop do
+        begin
+          receive_messages(&block)
+        rescue *CONNECTION_ERRORS => e
+          logger.error("Connection error to #{queue}: #{e}")
+          raise SubscribeError.new(e)
         end
       end
+    end
 
-      if async?
-        process_asynchronously(&process)
-      else
-        process.call
-      end
+    def self.async_strategies
+      super - [:batch]
     end
 
-    def async?
-      @async
+    def self.default_async_strategy
+      Circuitry.config.subscribe_async_strategy
     end
 
+    protected
+
+    attr_writer :queue, :wait_time, :batch_size
+
     private
 
     def receive_messages(&block)
@@ -70,7 +70,15 @@ module Circuitry
       return if messages.empty?
 
       messages.each do |message|
-        process_message(message, &block)
+        process = -> do
+          process_message(message, &block)
+        end
+
+        if async?
+          process_asynchronously(&process)
+        else
+          process.call
+        end
       end
     end
 
@@ -78,12 +86,12 @@ module Circuitry
       message = Message.new(message)
 
       unless message.nil?
-        logger.info "Processing message #{message.id}"
+        logger.info("Processing message #{message.id}")
         handle_message(message, &block)
         delete_message(message)
       end
     rescue => e
-      logger.error "Error processing message #{message.id}: #{e}"
+      logger.error("Error processing message #{message.id}: #{e}")
       error_handler.call(e) if error_handler
     end
 

data/lib/circuitry/version.rb

@@ -1,3 +1,3 @@
 module Circuitry
-  VERSION = '1.0.0'
+  VERSION = '1.1.0'
 end

data/lib/circuitry.rb

@@ -1,4 +1,8 @@
 require 'circuitry/version'
+require 'circuitry/processor'
+require 'circuitry/processors/batcher'
+require 'circuitry/processors/forker'
+require 'circuitry/processors/threader'
 require 'circuitry/configuration'
 require 'circuitry/publisher'
 require 'circuitry/subscriber'
@@ -18,7 +22,9 @@ module Circuitry
     Subscriber.new(queue, options).subscribe(&block)
   end
 
-  def self.platform_supports_async?
-    Process.respond_to?(:fork)
+  def self.flush
+    Processors.constants.each do |const|
+      Processors.const_get(const).flush
+    end
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: circuitry
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.1.0
 platform: ruby
 authors:
 - Matt Huggins
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2015-06-25 00:00:00.000000000 Z
+date: 2015-07-08 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fog-aws
@@ -143,6 +143,10 @@ files:
 - lib/circuitry/concerns/async.rb
 - lib/circuitry/configuration.rb
 - lib/circuitry/message.rb
+- lib/circuitry/processor.rb
+- lib/circuitry/processors/batcher.rb
+- lib/circuitry/processors/forker.rb
+- lib/circuitry/processors/threader.rb
 - lib/circuitry/publisher.rb
 - lib/circuitry/services/sns.rb
 - lib/circuitry/services/sqs.rb