circuitry 2.1.1 → 3.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: f96fa19eef4e73b058cf470c6551e07a5f478a97
-  data.tar.gz: faeadab02b2bbf5aa6f1ca6ed59932ca9463ffe5
+  metadata.gz: df4dbe3ed4201727d1176f9fe8a520ec273b70e6
+  data.tar.gz: ee091c7d8112aa937040c5bb9dbfb3b4979094ee
 SHA512:
-  metadata.gz: fcf9f9976723c3bd6abe59d18cabdd27a3730b48f409d120201ab22f537443f47fa2087e64d19429e1241000ad71a88a538e05bae9f94c5a5d3fc79975a9d52f
-  data.tar.gz: 45e85b9010fc0967df18b7a9c19645950ba2d70273949485457a7f8ec72cf1bc50a990bd69bfb504e333c2a040152107f11a5bb4dfb8eda27074c89031bcdfcc
+  metadata.gz: 963e72ac4514a4d5eaa65eff1a745bbb8c4fce67cb8d244e9287bdc9b5017d17eeb8b97a24555ae4be3764c880052320c9aa219f395af7ba88bdbcae5a35e3df
+  data.tar.gz: 5e42c1e6858ca016b19dc6c1f954547672e20ff9857d06d58747aa05f4266e84c46eb1c9f45159b12102ef7664cc6fa2ed12787440835c60fb9c2c67521622d6

data/CHANGELOG.md

@@ -1,3 +1,10 @@
+## Circuitry 3.0.0 (Feb 19, 2016)
+
+* Added separate configuration for publisher/subscriber applications *Brandon Croft*
+* Added YML config option *Brandon Croft*
+* Added max_receive_count and visibility_timeout subscriber config options *Brandon Croft*
+* Replace on_thread_exit and on_fork_exit with on_async_exit config option *Brandon Croft*
+
 ## Circuitry 2.1.1 (Jan 30, 2016)
 
 * Fixed missing require in subscriber *Brandon Croft*

data/README.md

@@ -29,10 +29,13 @@ Or install it yourself as:
 
 ## Usage
 
-Circuitry is configured via its configuration object.
+Circuitry is configured via its configuration object or via circuitry.yml config.
 
 ```ruby
-Circuitry.config do |c|
+Circuitry.subscriber_config do |c|
+  c.queue_name = "#{Rails.env}-appname"
+  c.dead_letter_queue_name = "#{Rails.env}-appname-failures"
+  c.max_receive_count = 8
   c.access_key = 'YOUR_AWS_ACCESS_KEY'
   c.secret_key = 'YOUR_AWS_SECRET_KEY'
   c.region = 'us-east-1'
@@ -42,14 +45,58 @@ Circuitry.config do |c|
     HoneyBadger.flush
   end
   c.lock_strategy = Circuitry::Locks::Redis.new(url: 'redis://localhost:6379')
-  c.publish_async_strategy = :batch
-  c.subscribe_async_strategy = :thread
-  c.on_thread_exit = proc { Mongoid.disconnect_sessions }
-  c.on_fork_exit = proc { Mongoid.disconnect_sessions }
+  c.async_strategy = :thread
+  c.on_async_exit = proc { Mongoid.disconnect_sessions }
+end
+
+Circuitry.publisher_config do |c|
+  c.access_key = 'YOUR_AWS_ACCESS_KEY'
+  c.secret_key = 'YOUR_AWS_SECRET_KEY'
+  c.region = 'us-east-1'
+  c.logger = Rails.logger
+  c.error_handler = proc do |error|
+    HoneyBadger.notify(error)
+    HoneyBadger.flush
+  end
+  c.async_strategy = :batch
 end
 ```
 
-Available configuration options include:
+Many of the advanced options, such as `error_handler` or `async_strategy` require this initializer-
+style configuration. A simpler option is available via config/circuitry.yml
+(or config/circuitry.yml.erb):
+
+```yml
+---
+access_key: "YOUR_AWS_ACCESS_KEY"
+secret_key: "YOUR_AWS_SECRET_KEY"
+region: "us-east-1"
+
+development:
+  publisher:
+    topic_names:
+      - brandonc-appname-user-create
+      - brandonc-appname-user-destroy
+  subscriber:
+    queue_name: "brandonc-appname"
+    dead_letter_queue_name: "brandonc-appname-failures"
+    topic_names:
+      - brandonc-otherapp-content-create
+      - brandonc-otherapp-content-destroy
+production:
+  publisher:
+    topic_names:
+      - brandonc-appname-user-create
+      - brandonc-appname-user-destroy
+  subscriber:
+    queue_name: "production-appname"
+    dead_letter_queue_name: "production-appname-failures"
+    topic_names:
+      - production-otherapp-content-create
+      - production-otherapp-content-destroy
+```
+
+Available configuration options for *both* subscriber and publisher applications include:
 
 * `access_key`: The AWS access key ID that has access to SNS publishing and/or
   SQS subscribing. *(required)*
@@ -62,10 +109,7 @@ Available configuration options include:
 * `error_handler`: An object that responds to `call` with two arguments: the
   deserialized message contents and the topic name used when publishing to SNS.
   *(optional, default: `nil`)*
-* `lock_strategy` - The store used to ensure that no duplicate messages are
-  processed. Please refer to the [Lock Strategies](#lock-strategies) section for
-  more details regarding this option. *(default: `Circuitry::Locks::Memory.new`)*
-* `publish_async_strategy`: One of `:fork`, `:thread`, or `:batch` that
+* `async_strategy`: One of `:fork`, `:thread`, or `:batch` that
   determines how asynchronous publish requests are processed. *(optional,
   default: `:fork`)*
   * `:fork`: Forks a detached child process that immediately sends the request.
@@ -73,36 +117,36 @@ Available configuration options include:
     threads are not guaranteed to complete when the process exits, completion can
     be ensured by calling `Circuitry.flush`.
   * `:batch`: Stores the request in memory to be submitted later. Batched
-    requests must be manually sent by calling `Circuitry.flush`.
-* `subscribe_async_strategy`: One of `:fork` or `:thread` that determines how
-  asynchronous subscribe requests are processed. *(optional, default: `:fork`)*
-  * `:fork`: Forks a detached child process that immediately begins querying the
-    queue.
-  * `:thread`: Creates a new thread that immediately sends begins querying the
-    queue.
-* `on_thread_exit`: An object that responds to `call`. This is useful for
+    requests must be manually sent by calling `Circuitry.flush`. Only valid as a
+    publishing strategy
+* `on_async_exit`: An object that responds to `call`. This is useful for
   managing shared resources such as database connections that require closing.
-  It is only called when implementing the `:thread` async strategy. *(optional,
-  default: `nil`)*
-* `on_fork_exit`: An object that responds to `call`. This is useful for
-  managing shared resources such as database connections that require closing,
-  It is only called when implementing the `:fork` async strategy. *(optional,
-  default: `nil`)*
-* `publisher_topic_names`: An array of topic names that your publishing application will
-  publish on. This configuration is only used during provisioning via `rake circuitry:setup`
-* `subscriber_queue_name`: The name of the SQS queue that your subscriber application
-  will listen to. This queue will be created or configured during `rake circuitry:setup`
   *(optional, default: `nil`)*
-* `subscriber_dead_letter_queue_name`: The name of the SQS dead letter queue that will be
-  used after all retries fail. This queue will be created and configured during `rake
-  circuitry:setup` *(optional, default: `<subscriber_queue_name>-failures`)*
-* `publisher_middleware`: A chain of middleware that sent messages must go through.
-  Please refer to the [Middleware](#middleware) section for more details regarding this
-  option.
-* `subscriber_middleware`: A chain of middleware that received messages must go through.
+* `topic_names`: An array of topic names that your application will
+  publish and/or subscribe to. This configuration is only used during provisioning.
+* `middleware`: A chain of middleware that messages must go through when sent or received.
   Please refer to the [Middleware](#middleware) section for more details regarding this
   option.
 
+Available configuration options for subscriber applications include:
+
+* `queue_name`: The name of the SQS queue that your subscriber application
+  will listen to. This queue will be created or configured during provisioning.
+* `dead_letter_queue_name`: The name of the SQS dead letter queue that will be
+  used after all retries fail. This configuration value is only used during provisioning.
+  *(optional, default: `<subscriber_queue_name>-failures`)*
+* `lock_strategy` - The store used to ensure that no duplicate messages are
+  processed. Please refer to the [Lock Strategies](#lock-strategies) section for
+  more details regarding this option. *(default: `Circuitry::Locks::Memory.new`)*
+* `max_receive_count` - The number of times a message will be received by the queue after
+  unsuccessful attempts to process it before it is discarded or added to the
+  `dead_letter_queue_name` queue. This configuration value is only used during
+  provisioning. *(optional, default: 8)*
+* `visibility_timeout` - A period of time during which Amazon SQS prevents other subscribers from
+  receiving and processing that message (before it is deleted by circuitry after being processed
+  successfully.) This configuration value is only used during provisioning.
+  *(optional, default: 1800)*
+
 ### Provisioning
 
 You can automatically provision SQS queues, SNS topics, and the subscriptions between them using
@@ -110,17 +154,16 @@ two methods: the circuitry CLI or the `rake circuitry:setup` task. The rake task
 subscriber queue and publishing topics that are configured within your application.
 
 ```ruby
-Circuitry.config do |c|
-  c.subscriber_queue_name = 'myapp-production-events'
-  c.publisher_topic_names = ['theirapp-production-stuff-created', 'theirapp-production-stuff-deleted']
+Circuitry.subscriber_config do |c|
+  c.queue_name = 'myapp-production-events'
+  c.topic_names = ['theirapp-production-stuff-created', 'theirapp-production-stuff-deleted']
 end
 ```
 
-When provisioning, a dead letter queue is also created using the name "<queue_name>-failures" and a
-redrive policy of 8 retries to that dead letter queue is configured. You can customize the dead
-letter queue name in your configuration.
+When provisioning, a dead letter queue is also created using the name "<queue_name>-failures". You
+can customize the dead letter queue name in your configuration.
 
-Run `ruby bin/circuitry help provision` for help using CLI provisioning.
+Run `circuitry help provision` for help using CLI provisioning.
 
 ### Publishing
 
@@ -139,7 +182,7 @@ The `publish` method also accepts options that impact instantiation of the
 
 * `:async` - Whether or not publishing should occur in the background. Accepts
   one of `:fork`, `:thread`, `:batch`, `true`, or `false`. Passing `true` uses
-  the `publish_async_strategy` value from the gem configuration. Please refer to
+  the `async_strategy` value from the gem configuration. Please refer to
   the [Asynchronous Support](#asynchronous-support) section for more details
   regarding this option. *(default: `false`)*
 * `:timeout` - The maximum amount of time in seconds that publishing a message
@@ -183,7 +226,7 @@ The `subscribe` method also accepts options that impact instantiation of the
   regarding this option. *(default: `true`)*
 * `:async` - Whether or not subscribing should occur in the background. Accepts
   one of `:fork`, `:thread`, `true`, or `false`. Passing `true` uses the
-  `subscribe_async_strategy` value from the gem configuration. Passing an
+  `async_strategy` value from the gem configuration. Passing an
   asynchronous value will cause messages to be handled concurrently. Please
   refer to the [Asynchronous Support](#asynchronous-support) section for more
   details regarding this option. *(default: `false`)*
@@ -206,7 +249,7 @@ options = {
   batch_size: 20
 }
 
-Circuitry.subscribe('https://...', options) do |message, topic_name|
+Circuitry.subscribe(options) do |message, topic_name|
   # ...
 end
 ```
@@ -217,7 +260,7 @@ Alternatively, if your options hash will remain unchanged, you can build a singl
 ```ruby
 options = { ... }
 subscriber = Circuitry::Subscriber.new(options)
-subscriber.subscribe('https://...') do |message, topic_name|
+subscriber.subscribe do |message, topic_name|
   # ...
 end
 ```
@@ -303,7 +346,7 @@ The soft and hard TTL values can be changed by passing a `:soft_ttl` or
 that a lock should persist. For example:
 
 ```ruby
-Circuitry.config.lock_strategy = Circuitry::Locks::Memory.new(
+Circuitry.subscriber_config.lock_strategy = Circuitry::Locks::Memory.new(
     soft_ttl: 10 * 60,      # 10 minutes
     hard_ttl: 48 * 60 * 60  # 48 hours
 )
@@ -430,7 +473,7 @@ pass your lock instance to the configuration as the `:lock_strategy`.
 
 ```ruby
 connection = PG.connect(...)
-Circuitry.config.lock_strategy = DatabaseLockStrategy.new(connection: connection)
+Circuitry.subcriber_config.lock_strategy = DatabaseLockStrategy.new(connection: connection)
 ```
 
 ### Middleware
@@ -454,7 +497,7 @@ class LoggerMiddleware
     self.namespace = namespace
     self.logger = logger
   end
-  
+
   def call(topic, message)
     logger.info("#{namespace} (start): #{topic} - #{message}")
     yield
@@ -471,24 +514,29 @@ end
 Adding the middleware to the stack happens through the Circuitry config.
 
 ```ruby
-Circuitry.config do |config|
+Circuitry.subscriber_config do |config|
   # single-line format
-  circuitry.publisher_middleware.add LoggerMiddleware, namespace: 'publisher'
-  circuitry.subscriber_middleware.add LoggerMiddleware, namespace: 'subscriber', logger: Rails.logger
+  config.middleware.add LoggerMiddleware, namespace: 'subscriber_app', logger: Rails.logger
 
   # block format
-  circuitry.publisher_middleware do |chain|
-    chain.add LoggerMiddleware, namespace: 'publisher'
+  config.middleware do |chain|
+    chain.add LoggerMiddleware, namespace: 'subscriber_app', logger: Rails.logger
   end
+end
 
-  circuitry.subscriber_middleware do |chain|
-    chain.add LoggerMiddleware, namespace: 'subscriber', logger: Rails.logger
+Circuitry.publisher_config do |config|
+  # single-line format
+  config.middleware.add LoggerMiddleware, namespace: 'publisher_app'
+
+  # block format
+  config.middleware do |chain|
+    chain.add LoggerMiddleware, namespace: 'publisher_app'
   end
 end
 ```
 
-Both `publisher_middleware` and `subscriber_middleware` respond to a handful of methods that can be
-used for configuring your middleware:
+`config.middleware` responds to a handful of methods that can be used for configuring
+your middleware:
 
 * `#add`: Appends a middleware class to the end of the chain.  If the class already exists, it is
   replaced.

data/circuitry.gemspec

@@ -9,8 +9,8 @@ Gem::Specification.new do |spec|
   spec.authors       = ['Matt Huggins', 'Brandon Croft']
   spec.email         = ['matt.huggins@kapost.com', 'brandon@kapost.com']
 
-  spec.summary       = %q{Decouple ruby applications using Amazon SNS fanout with SQS processing.}
-  spec.description   = %q{A Circuitry publisher application can broadcast events which can be processed independently by Circuitry subscriber applications.}
+  spec.summary       = 'Decouple ruby applications using Amazon SNS fanout with SQS processing.'
+  spec.description   = 'A Circuitry publisher application can broadcast events which can be processed independently by Circuitry subscriber applications.'
   spec.homepage      = 'https://github.com/kapost/circuitry'
   spec.license       = 'MIT'
 

data/lib/circuitry.rb

@@ -1,5 +1,6 @@
 require 'circuitry/railtie' if defined?(Rails) && Rails::VERSION::MAJOR >= 3
-require 'circuitry/configuration'
+require 'circuitry/config/publisher_settings'
+require 'circuitry/config/subscriber_settings'
 require 'circuitry/locks/base'
 require 'circuitry/locks/memcache'
 require 'circuitry/locks/memory'
@@ -15,23 +16,39 @@ require 'circuitry/subscriber'
 require 'circuitry/version'
 
 module Circuitry
-  def self.config(&block)
-    @config ||= Configuration.new
-    block.call(@config) if block_given?
-    @config
-  end
+  class << self
+    def subscriber_config
+      @_sub_config ||= Config::SubscriberSettings.new
+      yield @_sub_config if block_given?
+      @_sub_config
+    end
 
-  def self.publish(topic_name, object, options = {})
-    Publisher.new(options).publish(topic_name, object)
-  end
+    def subscriber_config=(options)
+      @_sub_config = Config::SubscriberSettings.new(options)
+    end
 
-  def self.subscribe(options = {}, &block)
-    Subscriber.new(options).subscribe(&block)
-  end
+    def publisher_config
+      @_pub_config ||= Config::PublisherSettings.new
+      yield @_pub_config if block_given?
+      @_pub_config
+    end
+
+    def publisher_config=(options)
+      @_pub_config = Config::PublisherSettings.new(options)
+    end
+
+    def publish(topic_name, object, options = {})
+      Publisher.new(options).publish(topic_name, object)
+    end
+
+    def subscribe(options = {}, &block)
+      Subscriber.new(options).subscribe(&block)
+    end
 
-  def self.flush
-    Processors.constants.each do |const|
-      Processors.const_get(const).flush
+    def flush
+      Processors.constants.each do |const|
+        Processors.const_get(const).flush
+      end
     end
   end
 end

data/lib/circuitry/cli.rb

@@ -1,11 +1,13 @@
-require 'circuitry/provisioner'
+require 'circuitry/provisioning'
 require 'thor'
 
 module Circuitry
   class CLI < Thor
     class_option :verbose, aliases: :v, type: :boolean
 
-    desc 'provision <queue> -t <topic> [<topic> ...]', 'Provision a queue subscribed to one or more topics'
+    desc 'provision <queue> -t <topic> [<topic> ...]', <<-END
+      Provision a queue subscribed to one or more topics
+    END
 
     long_desc <<-END
       Creates an SQS queue with appropriate SNS access policy along with one or more SNS topics
@@ -21,18 +23,25 @@ module Circuitry
       name <queue>-failures
     END
 
-    option :topics, aliases: :t, type: :array, required: :true
-    option :access_key, aliases: :a
-    option :secret_key, aliases: :s
-    option :dead_letter_queue, aliases: :d
-    option :region, aliases: :r
+    option :topic_names, aliases: :t, type: :array, required: true
+    option :access_key, aliases: :a, required: true
+    option :secret_key, aliases: :s, required: true
+    option :region, aliases: :r, required: true
+    option :dead_letter_queue_name, aliases: :d
+    option :visibility_timeout, aliases: :v, default: 30 * 60
+    option :max_receive_count, aliases: :n, default: 8
+
+    OPTIONS_KEYS_PUBLISHER_CONFIG = [:access_key, :secret_key, :region].freeze
+
+    OPTIONS_KEYS_SUBSCRIBER_CONFIG = [:access_key, :secret_key, :region, :dead_letter_queue_name,
+                                      :topic_names, :max_receive_count, :visibility_timeout].freeze
 
     def provision(queue_name)
-      with_custom_config(queue_name) do |config|
-        logger = Logger.new(STDOUT)
-        logger.level = Logger::INFO if options['verbose']
-        Circuitry::Provisioner.new(config, logger: logger).run
-      end
+      initialize_config(queue_name)
+
+      logger = Logger.new(STDOUT)
+      logger.level = Logger::INFO if options['verbose']
+      Circuitry::Provisioning.provision(logger: logger)
     end
 
     private
@@ -41,30 +50,21 @@ module Circuitry
       puts(*args) if options['verbose']
     end
 
-    def with_custom_config(queue_name, &block)
-      original_values = {}
-      %i[access_key secret_key region subscriber_queue_name subscriber_dead_letter_queue_name publisher_topic_names].each do |sym|
-        original_values[sym] = Circuitry.config.send(sym)
-      end
-
-      assign_options_config(queue_name, original_values)
+    def initialize_config(queue_name)
+      Circuitry.publisher_config.topic_names = []
+      Circuitry.subscriber_config.queue_name = queue_name
 
-      block.call(Circuitry.config)
-    ensure
-      restore_config(original_values)
+      assign_options_config
     end
 
-    def assign_options_config(queue_name, original_values)
-      Circuitry.config.access_key = options.fetch('access_key', original_values[:access_key])
-      Circuitry.config.secret_key = options.fetch('secret_key', original_values[:secret_key])
-      Circuitry.config.region = options.fetch('region', original_values[:region])
-      Circuitry.config.subscriber_queue_name = queue_name
-      Circuitry.config.subscriber_dead_letter_queue_name = options.fetch('dead_letter_queue', "#{queue_name}-failures")
-      Circuitry.config.publisher_topic_names = options['topics']
-    end
+    def assign_options_config
+      OPTIONS_KEYS_PUBLISHER_CONFIG.each do |key|
+        Circuitry.publisher_config.send(:"#{key}=", options[key.to_s])
+      end
 
-    def restore_config(original_values)
-      original_values.keys.each { |key| Circuitry.config.send(:"#{key}=", original_values[key]) }
+      OPTIONS_KEYS_SUBSCRIBER_CONFIG.each do |key|
+        Circuitry.subscriber_config.send(:"#{key}=", options[key.to_s])
+      end
     end
   end
 end