circuitry 1.1.0 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: e63a7c32a7c197c43ef244bdb5b3be5ae1544f39
-  data.tar.gz: f99bee8e6cc734e0ae334d31de01eb70851717e3
+  metadata.gz: aabb7e5d67924cf49e3d6efdd5f34c27d4d850f6
+  data.tar.gz: b63bad1d70be23cec5c666b699f8473b25a2d7cb
 SHA512:
-  metadata.gz: 4bb4da24b7d24c81b212c3c8f750d8bed4c85e8bde06f96e25f08c5d6291e1bed908948eec337caa193333e821cd02a01cfc94d5c86dcb3eeeddd24fab893da5
-  data.tar.gz: a54b2dc0d13eb5172e869163bef080e2a3095f09e3fa832ce6b79d0a1b508ec9fdf03610c4a98be83a3c7127b914ea82a398a935beb05d12f2e5891a3ed00933
+  metadata.gz: 01e354ad6cfaccaf54184d932714609fbad3e95c9f0a35137e756abe5448fc4fda5b7a8f52333d7dc7f05e4b80d4d2cb2c3a30b52e271e90cc70ddd2e6efd4ab
+  data.tar.gz: 146e463dc1f63ea887ec99ffbd350d9b28d9f3aff2d96fca12ce407c200a2f3a1480ce2b4be15747980f533ec00e2716854607ec53e46249b6a705e74d137bcf

data/Gemfile

@@ -2,3 +2,5 @@ source 'https://rubygems.org'
 
 # Specify your gem's dependencies in circuitry.gemspec
 gemspec
+
+gem 'memcache_mock', '0.0.14', github: 'mhuggins/MemcacheMock', branch: 'expiry-and-add'

data/README.md

@@ -36,6 +36,7 @@ Circuitry.config do |c|
     HoneyBadger.notify(error)
     HoneyBadger.flush
   end
+  c.lock_strategy = Circuitry::Lock::Redis.new(url: 'redis://localhost:6379')
   c.publish_async_strategy = :batch
   c.subscribe_async_strategy = :thread
 end
@@ -54,6 +55,9 @@ Available configuration options include:
 * `error_handler`: An object that responds to `call` with two arguments: the
   deserialized message contents and the topic name used when publishing to SNS.
   *(optional, default: `nil`)*
+* `:lock_strategy` - The store used to ensure that no duplicate messages are
+  processed.  Please refer to the [Lock Strategies](#lock-strategies) section for
+  more details regarding this option.  *(default: `Circuitry::Locks::Memory.new`)*
 * `publish_async_strategy`: One of `:fork`, `:thread`, or `:batch` that
   determines how asynchronous publish requests are processed.  *(optional,
   default: `:fork`)*
@@ -90,10 +94,14 @@ The `publish` method also accepts options that impact instantiation of the
   the `publish_async_strategy` value from the gem configuration.  Please refer to
   the [Asynchronous Support](#asynchronous-support) section for more details
   regarding this option.  *(default: `false`)*
+* `:timeout` - The maximum amount of time in seconds that publishing a message
+  will be attempted before giving up.  If the timeout is exceeded, an exception
+  will raised to be handled by your application or `error_handler`. *(default:
+  15)*
 
 ```ruby
 obj = { foo: 'foo', bar: 'bar' }
-Circuitry.publish('my-topic-name', obj, async: true)
+Circuitry.publish('my-topic-name', obj, async: true, timeout: 20)
 ```
 
 Alternatively, if your options hash will remain unchanged, you can build a single
@@ -120,13 +128,22 @@ end
 The `subscribe` method also accepts options that impact instantiation of the
 `Subscriber` object, which currently includes the following options.
 
+* `:lock` - The strategy used to ensure that no duplicate messages are processed.
+  Accepts `true`, `false`, or an instance of a class inheriting from
+  `Circuitry::Locks::Base`.  Passing `true` uses the `lock_strategy` value from
+  the gem configuration.  Passing `false` uses the [NOOP](#NOOP) strategy. Please
+  refer to the [Lock Strategies](#lock-strategies) section for more details
+  regarding this option.  *(default: `true`)*
 * `:async` - Whether or not subscribing should occur in the background.  Accepts
   one of `:fork`, `:thread`, `true`, or `false`.  Passing `true` uses the
   `subscribe_async_strategy` value from the gem configuration.  Passing an
-  asynchronous value will cause messages to be handled concurrently, meaning
-  that queued messages *might not be processed in the order they're received*.
-  Please refer to the [Asynchronous Support](#asynchronous-support) section for
-  more details regarding this option.  *(default: `false`)*
+  asynchronous value will cause messages to be handled concurrently.  Please
+  refer to the [Asynchronous Support](#asynchronous-support) section for more
+  details regarding this option.  *(default: `false`)*
+* `:timeout` - The maximum amount of time in seconds that processing a message
+  will be attempted before giving up.  If the timeout is exceeded, an exception
+  will raised to be handled by your application or `error_handler`.  *(default:
+  15)*
 * `:wait_time` - The number of seconds to wait for messages while connected to
   SQS.  Anything above 0 results in long-polling, while 0 results in
   short-polling.  *(default: 10)*
@@ -134,7 +151,15 @@ The `subscribe` method also accepts options that impact instantiation of the
   *(default: 10)*
 
 ```ruby
-Circuitry.subscribe('https://...', async: true, wait_time: 60, batch_size: 20) do |message, topic_name|
+options = {
+  lock: true,
+  async: true,
+  timeout: 20,
+  wait_time: 60,
+  batch_size: 20
+}
+
+Circuitry.subscribe('https://...', options) do |message, topic_name|
   # ...
 end
 ```
@@ -204,6 +229,152 @@ Batched publish and subscribe requests are queued in memory and do not begin
 processing until you explicit flush them.  This can be done by calling
 `Circuitry.flush`.
 
+### Lock Strategies
+
+The [Amazon SQS FAQ](http://aws.amazon.com/sqs/faqs/) includes the following
+important point:
+
+> Amazon SQS is engineered to provide “at least once” delivery of all messages in
+> its queues. Although most of the time each message will be delivered to your
+> application exactly once, you should design your system so that processing a
+> message more than once does not create any errors or inconsistencies.
+
+Given this, it's up to the user to ensure messages are not processed multiple
+times in the off chance that Amazon does not recognize that a message has been
+processed.
+
+The circuitry gem handles this by caching SQS message IDs: first via a "soft
+lock" that denotes the message is about to be processed, then via a "hard lock"
+that denotes the message has finished processing.
+
+The soft lock has a default TTL of 15 minutes (a seemingly sane amount of time
+during which processing most queue messages should certainly be able to
+complete), while the hard lock has a default TTL of 24 hours (based upon
+[a suggestion by an AWS employee](https://forums.aws.amazon.com/thread.jspa?threadID=140782#507605)).
+The soft and hard TTL values can be changed by passing a `:soft_ttl` or
+`:hard_ttl` value to the lock initializer, representing the number of seconds
+that a lock should persist.  For example:
+
+```ruby
+Circuitry.config.lock_strategy = Circuitry::Lock::Memory.new(
+    soft_ttl: 10 * 60,      # 10 minutes
+    hard_ttl: 48 * 60 * 60  # 48 hours
+)
+```
+
+#### Memory
+
+If not specified in your circuitry configuration, the memory store will be used
+by default.  This lock strategy is provided as the lowest barrier to entry given
+that it has no third-party dependencies.  It should be avoided if running
+multiple subscriber processes or if expecting a high throughput that would result
+in a large amount of memory consumption.
+
+```ruby
+Circuitry::Lock::Memory.new
+```
+
+#### Redis
+
+Using the redis lock strategy requires that you add `gem 'redis'` to your
+`Gemfile`, as it is not included bundled with the circuitry gem by default.
+
+There are two ways to use the redis lock strategy.  The first is to pass your
+redis connection options to the lock in the same way that you would when building
+a new `Redis` object.
+
+```ruby
+Circuitry::Lock::Redis.new(url: 'redis://localhost:6379')
+```
+
+The second way is to pass in a `:client` option that specifies the redis client
+itself.  This is useful for more advanced usage such as sharing an existing redis
+connection, utilizing [Redis::Namespace](https://github.com/resque/redis-namespace),
+or utilizing [hiredis](https://github.com/redis/hiredis-rb).
+
+```ruby
+client = Redis.new(url: 'redis://localhost:6379')
+Circuitry::Lock::Redis.new(client: client)
+```
+
+#### Memcache
+
+Using the memcache lock strategy requires that you add `gem 'dalli'` to your
+`Gemfile`, as it is not included bundled with the circuitry gem by default.
+
+There are two ways to use the memcache lock strategy.  The first is to pass your
+dalli connection host and options to the lock in the same way that you would when
+building a new `Dalli::Client` object.  The special `host` option will be treated
+as the memcache host, just as the first argument to `Dalli::Client`.
+
+```ruby
+Circuitry::Lock::Memcache.new(host: 'localhost:11211', namespace: '...')
+```
+
+The second way is to pass in a `:client` option that specifies the dalli client
+itself.  This is useful for sharing an existing memcache connection.
+
+```ruby
+client = Dalli::Client.new('localhost:11211', namespace: '...')
+Circuitry::Lock::Memcache.new(client: client)
+```
+
+#### NOOP
+
+Using the noop lock strategy permits you to continue to treat SQS as a
+distributed queue in a true sense, meaning that you might receive duplicate
+messages.  Please refer to the Amazon SQS documentation pertaining to the
+[Properties of Distributed Queues](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/DistributedQueues.html).
+
+#### Custom
+
+It's also possible to roll your own lock strategy.  Simply create a class that
+includes (or module that extends) `Circuitry::Lock::Base` and implements the
+following methods:
+
+* `lock`: Accepts the `key` and `ttl` as parameters.  If the key is already
+  locked, this method must return false.  If the key is not already locked, it
+  must lock the key for `ttl` seconds and return true.  It is important that
+  the check and update are **atomic** in order to ensure the same message isn't
+  processed more than once.
+* `lock!`: Accepts the `key` and `ttl` as parameters.  Must lock the key for
+  `ttl` seconds regardless of whether or not the key was previously locked.
+
+For example, a database-backed solution might look something like the following:
+
+```ruby
+class DatabaseLockStrategy
+  include Circuitry::Lock::Base
+
+  def initialize(options = {})
+    super(options)
+    self.connection = options.fetch(:connection)
+  end
+
+  protected
+
+  def lock(key, ttl)
+    connection.exec("INSERT INTO locks (key, expires_at) VALUES ('#{key}', '#{Time.now + ttl}')")
+  end
+
+  def lock!(key, ttl)
+    connection.exec("UPSERT INTO locks (key, expires_at) VALUES ('#{key}', '#{Time.now + ttl}')")
+  end
+
+  private
+
+  attr_reader :connection
+end
+```
+
+To use, simply create an instance of the class with your necessary options, and
+pass your lock instance to the configuration as the `:lock_strategy`.
+
+```ruby
+connection = PG.connect(...)
+Circuitry.config.lock_strategy = DatabaseLockStrategy.new(connection: connection)
+```
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run

data/circuitry.gemspec

@@ -22,10 +22,14 @@ Gem::Specification.new do |spec|
   spec.add_dependency 'fog-aws', '~> 0.4'
   spec.add_dependency 'virtus', '~> 1.0'
 
-  spec.add_development_dependency 'pry-nav'
+  spec.add_development_dependency 'pry-byebug'
   spec.add_development_dependency 'bundler', '~> 1.8'
   spec.add_development_dependency 'rake', '~> 10.0'
   spec.add_development_dependency 'rspec', '~> 3.2'
   spec.add_development_dependency 'rspec-its', '~> 1.2'
   spec.add_development_dependency 'codeclimate-test-reporter'
+  spec.add_development_dependency 'redis'
+  spec.add_development_dependency 'mock_redis'
+  spec.add_development_dependency 'dalli'
+  spec.add_development_dependency 'memcache_mock'
 end

data/lib/circuitry/configuration.rb

@@ -10,6 +10,7 @@ module Circuitry
     attribute :region, String, default: 'us-east-1'
     attribute :logger, Logger, default: Logger.new(STDERR)
     attribute :error_handler
+    attribute :lock_strategy, Object, default: ->(page, attribute) { Circuitry::Locks::Memory.new }
     attribute :publish_async_strategy, Symbol, default: ->(page, attribute) { :fork }
     attribute :subscribe_async_strategy, Symbol, default: ->(page, attribute) { :fork }
 

data/lib/circuitry/locks/base.rb

@@ -0,0 +1,41 @@
+module Circuitry
+  module Locks
+    module Base
+      DEFAULT_SOFT_TTL = (15 * 60).freeze       # 15 minutes
+      DEFAULT_HARD_TTL = (24 * 60 * 60).freeze  # 24 hours
+
+      attr_reader :soft_ttl, :hard_ttl
+
+      def initialize(options = {})
+        self.soft_ttl = options.fetch(:soft_ttl, DEFAULT_SOFT_TTL)
+        self.hard_ttl = options.fetch(:hard_ttl, DEFAULT_HARD_TTL)
+      end
+
+      def soft_lock(id)
+        lock(lock_key(id), soft_ttl)
+      end
+
+      def hard_lock(id)
+        lock!(lock_key(id), hard_ttl)
+      end
+
+      protected
+
+      def lock(key, ttl)
+        raise NotImplementedError
+      end
+
+      def lock!(key, ttl)
+        raise NotImplementedError
+      end
+
+      private
+
+      attr_writer :soft_ttl, :hard_ttl
+
+      def lock_key(id)
+        "circuitry:lock:#{id}"
+      end
+    end
+  end
+end

data/lib/circuitry/locks/memcache.rb

@@ -0,0 +1,30 @@
+module Circuitry
+  module Locks
+    class Memcache
+      include Base
+
+      def initialize(options = {})
+        super(options)
+
+        self.client = options.fetch(:client) do
+          require 'dalli'
+          ::Dalli::Client.new(options[:host], options)
+        end
+      end
+
+      protected
+
+      def lock(key, ttl)
+        client.add(key, (Time.now + ttl).to_i, ttl)
+      end
+
+      def lock!(key, ttl)
+        client.set(key, (Time.now + ttl).to_i, ttl)
+      end
+
+      private
+
+      attr_accessor :client
+    end
+  end
+end

data/lib/circuitry/locks/memory.rb

@@ -0,0 +1,59 @@
+module Circuitry
+  module Locks
+    class Memory
+      include Base
+
+      class << self
+        def store
+          @store ||= {}
+        end
+
+        def semaphore
+          @semaphore ||= Mutex.new
+        end
+      end
+
+      protected
+
+      def lock(key, ttl)
+        reap
+
+        store do |store|
+          if store.has_key?(key)
+            false
+          else
+            store[key] = Time.now + ttl
+            true
+          end
+        end
+      end
+
+      def lock!(key, ttl)
+        reap
+
+        store do |store|
+          store[key] = Time.now + ttl
+        end
+      end
+
+      private
+
+      def store(&block)
+        semaphore.synchronize do
+          block.call(self.class.store)
+        end
+      end
+
+      def reap
+        store do |store|
+          now = Time.now
+          store.delete_if { |key, expires_at| expires_at <= now }
+        end
+      end
+
+      def semaphore
+        self.class.semaphore
+      end
+    end
+  end
+end

data/lib/circuitry/locks/noop.rb

@@ -0,0 +1,17 @@
+module Circuitry
+  module Locks
+    class NOOP
+      include Base
+
+      protected
+
+      def lock(key, ttl)
+        true
+      end
+
+      def lock!(key, ttl)
+        # do nothing
+      end
+    end
+  end
+end

data/lib/circuitry/locks/redis.rb

@@ -0,0 +1,30 @@
+module Circuitry
+  module Locks
+    class Redis
+      include Base
+
+      def initialize(options = {})
+        super(options)
+
+        self.client = options.fetch(:client) do
+          require 'redis'
+          ::Redis.new(options)
+        end
+      end
+
+      protected
+
+      def lock(key, ttl)
+        client.set(key, (Time.now + ttl).to_i, ex: ttl, nx: true)
+      end
+
+      def lock!(key, ttl)
+        client.set(key, (Time.now + ttl).to_i, ex: ttl)
+      end
+
+      private
+
+      attr_accessor :client
+    end
+  end
+end

data/lib/circuitry/processor.rb

@@ -1,6 +1,6 @@
 module Circuitry
   module Processor
-    def process
+    def process(&block)
       raise NotImplementedError, "#{self} must implement class method `process`"
     end
 

data/lib/circuitry/publisher.rb

@@ -1,4 +1,5 @@
 require 'json'
+require 'timeout'
 require 'circuitry/concerns/async'
 require 'circuitry/services/sns'
 require 'circuitry/topic_creator'
@@ -12,12 +13,16 @@ module Circuitry
 
     DEFAULT_OPTIONS = {
         async: false,
+        timeout: 15,
     }.freeze
 
+    attr_reader :timeout
+
     def initialize(options = {})
       options = DEFAULT_OPTIONS.merge(options)
 
       self.async = options[:async]
+      self.timeout = options[:timeout]
     end
 
     def publish(topic_name, object)
@@ -30,8 +35,10 @@ module Circuitry
       end
 
       process = -> do
-        topic = TopicCreator.find_or_create(topic_name)
-        sns.publish(topic.arn, object.to_json)
+        Timeout.timeout(timeout) do
+          topic = TopicCreator.find_or_create(topic_name)
+          sns.publish(topic.arn, object.to_json)
+        end
       end
 
       if async?
@@ -45,6 +52,10 @@ module Circuitry
       Circuitry.config.publish_async_strategy
     end
 
+    protected
+
+    attr_writer :timeout
+
     private
 
     def logger

data/lib/circuitry/subscriber.rb

@@ -1,3 +1,4 @@
+require 'timeout'
 require 'circuitry/concerns/async'
 require 'circuitry/services/sqs'
 require 'circuitry/message'
@@ -9,10 +10,12 @@ module Circuitry
     include Concerns::Async
     include Services::SQS
 
-    attr_reader :queue, :wait_time, :batch_size, :max_retries, :failure_queue
+    attr_reader :queue, :timeout, :wait_time, :batch_size, :lock
 
     DEFAULT_OPTIONS = {
+        lock: true,
         async: false,
+        timeout: 15,
         wait_time: 10,
         batch_size: 10,
     }.freeze
@@ -27,7 +30,9 @@ module Circuitry
       options = DEFAULT_OPTIONS.merge(options)
 
       self.queue = queue
+      self.lock = options[:lock]
       self.async = options[:async]
+      self.timeout = options[:timeout]
       self.wait_time = options[:wait_time]
       self.batch_size = options[:batch_size]
     end
@@ -60,7 +65,18 @@ module Circuitry
 
     protected
 
-    attr_writer :queue, :wait_time, :batch_size
+    attr_writer :queue, :timeout, :wait_time, :batch_size
+
+    def lock=(value)
+      value = case value
+        when true then Circuitry.config.lock_strategy
+        when false then Circuitry::Locks::NOOP.new
+        when Circuitry::Locks::Base then value
+        else raise ArgumentError, "Invalid value `#{value}`, must be one of `true`, `false`, or instance of `#{Circuitry::Locks::Base}`"
+      end
+
+      @lock = value
+    end
 
     private
 
@@ -85,21 +101,29 @@ module Circuitry
     def process_message(message, &block)
       message = Message.new(message)
 
-      unless message.nil?
-        logger.info("Processing message #{message.id}")
-        handle_message(message, &block)
-        delete_message(message)
-      end
+      logger.info("Processing message #{message.id}")
+      handle_message(message, &block)
+      delete_message(message)
     rescue => e
       logger.error("Error processing message #{message.id}: #{e}")
       error_handler.call(e) if error_handler
     end
 
     def handle_message(message, &block)
-      block.call(message.body, message.topic.name)
-    rescue => e
-      logger.error("Error handling message #{message.id}: #{e}")
-      raise e
+      if lock.soft_lock(message.id)
+        begin
+          Timeout.timeout(timeout) do
+            block.call(message.body, message.topic.name)
+          end
+        rescue => e
+          logger.error("Error handling message #{message.id}: #{e}")
+          raise e
+        end
+
+        lock.hard_lock(message.id)
+      else
+        logger.info("Ignoring duplicate message #{message.id}")
+      end
     end
 
     def delete_message(message)

data/lib/circuitry/version.rb

@@ -1,3 +1,3 @@
 module Circuitry
-  VERSION = '1.1.0'
+  VERSION = '1.2.0'
 end

data/lib/circuitry.rb

@@ -1,4 +1,9 @@
 require 'circuitry/version'
+require 'circuitry/locks/base'
+require 'circuitry/locks/memcache'
+require 'circuitry/locks/memory'
+require 'circuitry/locks/noop'
+require 'circuitry/locks/redis'
 require 'circuitry/processor'
 require 'circuitry/processors/batcher'
 require 'circuitry/processors/forker'

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: circuitry
 version: !ruby/object:Gem::Version
-  version: 1.1.0
+  version: 1.2.0
 platform: ruby
 authors:
 - Matt Huggins
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2015-07-08 00:00:00.000000000 Z
+date: 2015-07-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fog-aws
@@ -39,7 +39,7 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '1.0'
 - !ruby/object:Gem::Dependency
-  name: pry-nav
+  name: pry-byebug
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -122,6 +122,62 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: redis
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: mock_redis
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: dalli
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: memcache_mock
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: Amazon SNS publishing and SQS queue processing.
 email:
 - matt.huggins@kapost.com
@@ -142,6 +198,11 @@ files:
 - lib/circuitry.rb
 - lib/circuitry/concerns/async.rb
 - lib/circuitry/configuration.rb
+- lib/circuitry/locks/base.rb
+- lib/circuitry/locks/memcache.rb
+- lib/circuitry/locks/memory.rb
+- lib/circuitry/locks/noop.rb
+- lib/circuitry/locks/redis.rb
 - lib/circuitry/message.rb
 - lib/circuitry/processor.rb
 - lib/circuitry/processors/batcher.rb