circuitry 2.0.0 → 2.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 7d3f37801853268fc7adb65ff0f4d4c763f0f098
4
- data.tar.gz: aae27a8ce4d592ff7e817c0aee62e2c82a973026
3
+ metadata.gz: 14368085a767ad616196eb22569c3c981af716d0
4
+ data.tar.gz: 639f43f5b3e192241e5b142c5251331ee8f86362
5
5
  SHA512:
6
- metadata.gz: bdd6ebe7c82eec7d98dafbfad50cd056c554db06a9fbd4f41d749c3ed94bfa18efafbd693404ca3863804e007fa941807acee55376db52dbba2e0abb99457d5a
7
- data.tar.gz: 01b6255d1c8d3826650cb4d7a263b71c28d310e7a69fb207b1e322495741531e36b21bd5d263f5702665a328077b6c7a5d19f3a4d6ca872d3162bf170d3f5ed7
6
+ metadata.gz: 15d2f7452c6f1324850b6b83a825ecc9c5bc7895319660b785917625dbf0a991f92ce7f4303c239df4eb520b8c128e427fd69c85069e687986fdc198d693266b
7
+ data.tar.gz: 8b4073842d2dca9813ee809ffcbab89ce8949ba148edbfa7d32531be5a79764eccc4f067489decbccccd143f86afa7db8767c55b34b9be64337ca9bc18aa8b41
data/CHANGELOG.md CHANGED
@@ -1,9 +1,13 @@
1
- ## Circuitry 2.0.0 (Jan 27, 2016)
1
+ ## Circuitry 2.1.0 (Jan 28, 2016)
2
2
 
3
- * Added subscriber_queue_name config *Brandon Croft*
4
- * Added publisher_topic_names config *Brandon Croft*
5
- * Added CLI and rake provisioning of queues and topics as defined by config *Brandon Croft*
6
- * Removed the requirement to provide a SQS URL to the subscriber *Brandon Croft*
3
+ * Added publisher and subscriber middleware. *Matt Huggins*
4
+
5
+ ## Circuitry 2.0.0 (Jan 28, 2016)
6
+
7
+ * Added subscriber_queue_name config. *Brandon Croft*
8
+ * Added publisher_topic_names config. *Brandon Croft*
9
+ * Added CLI and rake provisioning of queues and topics as defined by config. *Brandon Croft*
10
+ * Removed the requirement to provide a SQS URL to the subscriber. *Brandon Croft*
7
11
 
8
12
  ## Circuitry 1.4.1 (Jan 21, 2016)
9
13
 
data/README.md CHANGED
@@ -87,18 +87,26 @@ Available configuration options include:
87
87
  managing shared resources such as database connections that require closing,
88
88
  It is only called when implementing the `:fork` async strategy. *(optional,
89
89
  default: `nil`)*
90
+ * `publisher_topic_names`: An array of topic names that your publishing application will
91
+ publish on. This configuration is only used during provisioning via `rake circuitry:setup`
90
92
  * `subscriber_queue_name`: The name of the SQS queue that your subscriber application
91
93
  will listen to. This queue will be created or configured during `rake circuitry:setup`
92
94
  *(optional, default: `nil`)*
93
95
  * `subscriber_dead_letter_queue_name`: The name of the SQS dead letter queue that will be
94
96
  used after all retries fail. This queue will be created and configured during `rake
95
97
  circuitry:setup` *(optional, default: `<subscriber_queue_name>-failures`)*
96
- * `publisher_topic_names`: An array of topic names that your publishing application will
97
- publish on. This configuration is only used during provisioning via `rake circuitry:setup`
98
+ * `publisher_middleware`: A chain of middleware that sent messages must go through.
99
+ Please refer to the [Middleware](#middleware) section for more details regarding this
100
+ option.
101
+ * `subscriber_middleware`: A chain of middleware that received messages must go through.
102
+ Please refer to the [Middleware](#middleware) section for more details regarding this
103
+ option.
98
104
 
99
105
  ### Provisioning
100
106
 
101
- You can automatically provision SQS queues, SNS topics, and the subscriptions between them using two methods: the circuitry CLI or the `rake circuitry:setup` task. The rake task will provision the subscriber queue and publishing topics that are configured within your application.
107
+ You can automatically provision SQS queues, SNS topics, and the subscriptions between them using
108
+ two methods: the circuitry CLI or the `rake circuitry:setup` task. The rake task will provision the
109
+ subscriber queue and publishing topics that are configured within your application.
102
110
 
103
111
  ```ruby
104
112
  Circuitry.config do |c|
@@ -107,7 +115,9 @@ Circuitry.config do |c|
107
115
  end
108
116
  ```
109
117
 
110
- When provisioning, a dead letter queue is also created using the name "<queue_name>-failures" and a redrive policy of 8 retries to that dead letter queue is configured. You can customize the dead letter queue name in your configuration.
118
+ When provisioning, a dead letter queue is also created using the name "<queue_name>-failures" and a
119
+ redrive policy of 8 retries to that dead letter queue is configured. You can customize the dead
120
+ letter queue name in your configuration.
111
121
 
112
122
  Run `ruby bin/circuitry help provision` for help using CLI provisioning.
113
123
 
@@ -152,8 +162,8 @@ publisher.publish('my-topic-name', obj)
152
162
 
153
163
  ### Subscribing
154
164
 
155
- Subscribing is done via the `Circuitry.subscribe` method. It accepts a block for processing each message. This method **indefinitely
156
- blocks**, processing messages as they are enqueued.
165
+ Subscribing is done via the `Circuitry.subscribe` method. It accepts a block for processing each
166
+ message. This method **indefinitely blocks**, processing messages as they are enqueued.
157
167
 
158
168
  ```ruby
159
169
  Circuitry.subscribe do |message, topic_name|
@@ -422,6 +432,80 @@ connection = PG.connect(...)
422
432
  Circuitry.config.lock_strategy = DatabaseLockStrategy.new(connection: connection)
423
433
  ```
424
434
 
435
+ ### Middleware
436
+
437
+ Circuitry middleware can be used to perform additional processing around a message
438
+ being sent by a publisher or received by a subscriber. Some examples of processing
439
+ that belong here are monitoring or encryption specific to your application.
440
+
441
+ Middleware can be added to the publisher, the subscriber, or both. A middleware
442
+ class is defined by an (optional) `#initialize` method that accepts any number of
443
+ arguments, as well as a `#call` method that accepts the `topic` string, `message`
444
+ string, and a block for continuing processing.
445
+
446
+ For example, a simple logging middleware might look something like the following:
447
+
448
+ ```ruby
449
+ class LoggerMiddleware
450
+ attr_reader :namespace, :logger
451
+
452
+ def initialize(namespace:, logger: Logger.new(STDOUT))
453
+ self.namespace = namespace
454
+ self.logger = logger
455
+ end
456
+
457
+ def call(topic, message)
458
+ logger.info("#{namespace} (start): #{topic} - #{message}")
459
+ yield
460
+ ensure
461
+ logger.info("#{namespace} (done): #{topic} - #{message}")
462
+ end
463
+
464
+ private
465
+
466
+ attr_writer :namespace, :logger
467
+ end
468
+ ```
469
+
470
+ Adding the middleware to the stack happens through the Circuitry config.
471
+
472
+ ```ruby
473
+ Circuitry.config do |config|
474
+ # single-line format
475
+ circuitry.publisher_middleware.add LoggerMiddleware, namespace: 'publisher'
476
+ circuitry.subscriber_middleware.add LoggerMiddleware, namespace: 'subscriber', logger: Rails.logger
477
+
478
+ # block format
479
+ circuitry.publisher_middleware do |chain|
480
+ chain.add LoggerMiddleware, namespace: 'publisher'
481
+ end
482
+
483
+ circuitry.subscriber_middleware do |chain|
484
+ chain.add LoggerMiddleware, namespace: 'subscriber', logger: Rails.logger
485
+ end
486
+ end
487
+ ```
488
+
489
+ Both `publisher_middleware` and `subscriber_middleware` respond to a handful of methods that can be
490
+ used for configuring your middleware:
491
+
492
+ * `#add`: Appends a middleware class to the end of the chain. If the class already exists, it is
493
+ replaced.
494
+ * `middleware.add NewMiddleware, arg1, arg2, ...`
495
+ * `#prepend`: Prepends a middleware class to the beginning of the chain. If the class already
496
+ exists, it is replaced.
497
+ * `middleware.prepend NewMiddleware, arg1, arg2, ...`
498
+ * `#remove`: Removes a middleware class from anywhere in the chain.
499
+ * `middleware.remove NewMiddleware`
500
+ * `#insert_before`: Injects a middleware class before another middleware class in the chain. If
501
+ the other class does not exist in the chain, this behaves the same as `#prepend`.
502
+ * `middleware.insert_before ExistingMiddleware, NewMiddleware, arg1, arg2...`
503
+ * `#insert_after`: Injects a middleware class after another middleware class in the chain. If the
504
+ other class does not exist in the chain, this behaves the same as `#add`.
505
+ * `middleware.insert_after ExistingMiddleware, NewMiddleware, arg1, arg2...`
506
+ * `#clear`: Removes all middleware classes from the chain.
507
+ * `middleware.clear`
508
+
425
509
  ## Development
426
510
 
427
511
  After checking out the repo, run `bin/setup` to install dependencies. Then, run
data/lib/circuitry.rb CHANGED
@@ -5,6 +5,7 @@ require 'circuitry/locks/memcache'
5
5
  require 'circuitry/locks/memory'
6
6
  require 'circuitry/locks/noop'
7
7
  require 'circuitry/locks/redis'
8
+ require 'circuitry/middleware/chain'
8
9
  require 'circuitry/processor'
9
10
  require 'circuitry/processors/batcher'
10
11
  require 'circuitry/processors/forker'
@@ -34,6 +34,18 @@ module Circuitry
34
34
  super || "#{subscriber_queue_name}-failures"
35
35
  end
36
36
 
37
+ def subscriber_middleware
38
+ @subscriber_middleware ||= Circuitry::Middleware::Chain.new
39
+ yield @subscriber_middleware if block_given?
40
+ @subscriber_middleware
41
+ end
42
+
43
+ def publisher_middleware
44
+ @publisher_middleware ||= Circuitry::Middleware::Chain.new
45
+ yield @publisher_middleware if block_given?
46
+ @publisher_middleware
47
+ end
48
+
37
49
  def aws_options
38
50
  {
39
51
  access_key_id: access_key,
@@ -0,0 +1,82 @@
1
+ require 'circuitry/middleware/entry'
2
+
3
+ module Circuitry
4
+ module Middleware
5
+ class Chain
6
+ include Enumerable
7
+
8
+ def each(&block)
9
+ entries.each(&block)
10
+ end
11
+
12
+ def entries
13
+ @entries ||= []
14
+ end
15
+
16
+ def add(klass, *args)
17
+ remove(klass) if exists?(klass)
18
+ entries << Entry.new(klass, *args)
19
+ end
20
+
21
+ def remove(klass)
22
+ entries.delete_if { |entry| entry.klass == klass }
23
+ end
24
+
25
+ def prepend(klass, *args)
26
+ remove(klass) if exists?(klass)
27
+ entries.unshift(Entry.new(klass, *args))
28
+ end
29
+
30
+ def insert_before(old_klass, new_klass, *args)
31
+ new_entry = build_or_replace_entry(new_klass, *args)
32
+ i = entries.index { |entry| entry.klass == old_klass } || 0
33
+ entries.insert(i, new_entry)
34
+ end
35
+
36
+ def insert_after(old_klass, new_klass, *args)
37
+ new_entry = build_or_replace_entry(new_klass, *args)
38
+ i = entries.index { |entry| entry.klass == old_klass } || entries.size - 1
39
+ entries.insert(i + 1, new_entry)
40
+ end
41
+
42
+ def exists?(klass)
43
+ any? { |entry| entry.klass == klass }
44
+ end
45
+
46
+ def build
47
+ map(&:build)
48
+ end
49
+
50
+ def clear
51
+ entries.clear
52
+ end
53
+
54
+ def invoke(*args)
55
+ chain = build.dup
56
+
57
+ traverse_chain = -> do
58
+ if chain.empty?
59
+ yield
60
+ else
61
+ chain.shift.call(*args, &traverse_chain)
62
+ end
63
+ end
64
+
65
+ traverse_chain.call
66
+ end
67
+
68
+ private
69
+
70
+ def build_or_replace_entry(klass, *args)
71
+ i = entries.index { |entry| entry.klass == klass }
72
+ entry = i.nil? ? Entry.new(klass, *args) : entries.delete_at(i)
73
+
74
+ if entry.args == args
75
+ entry
76
+ else
77
+ Entry.new(klass, *args)
78
+ end
79
+ end
80
+ end
81
+ end
82
+ end
@@ -0,0 +1,20 @@
1
+ module Circuitry
2
+ module Middleware
3
+ class Entry
4
+ attr_reader :klass, :args
5
+
6
+ def initialize(klass, *args)
7
+ self.klass = klass
8
+ self.args = args
9
+ end
10
+
11
+ def build
12
+ klass.new(*args)
13
+ end
14
+
15
+ private
16
+
17
+ attr_writer :klass, :args
18
+ end
19
+ end
20
+ end
@@ -4,12 +4,11 @@ require 'circuitry/subscription_creator'
4
4
 
5
5
  module Circuitry
6
6
  class Provisioner
7
- attr_reader :log
8
- attr_reader :config
7
+ attr_reader :config, :logger
9
8
 
10
9
  def initialize(config, logger: Logger.new(STDOUT))
11
- @config = config
12
- @log = logger
10
+ self.config = config
11
+ self.logger = logger
13
12
  end
14
13
 
15
14
  def run
@@ -20,13 +19,15 @@ module Circuitry
20
19
 
21
20
  private
22
21
 
22
+ attr_writer :config, :logger
23
+
23
24
  def create_queue
24
25
  safe_aws('Create Queue') do
25
26
  queue = Circuitry::QueueCreator.find_or_create(
26
27
  config.subscriber_queue_name,
27
28
  dead_letter_queue_name: config.subscriber_dead_letter_queue_name
28
29
  )
29
- log.info "Created queue #{queue.url}"
30
+ logger.info "Created queue #{queue.url}"
30
31
  queue
31
32
  end
32
33
  end
@@ -35,7 +36,7 @@ module Circuitry
35
36
  safe_aws('Create Topics') do
36
37
  config.publisher_topic_names.map do |topic_name|
37
38
  topic = Circuitry::TopicCreator.find_or_create(topic_name)
38
- log.info "Created topic #{topic.name}"
39
+ logger.info "Created topic #{topic.name}"
39
40
  topic
40
41
  end
41
42
  end
@@ -44,7 +45,7 @@ module Circuitry
44
45
  def subscribe_topics(queue, topics)
45
46
  safe_aws('Subscribe Topics') do
46
47
  Circuitry::SubscriptionCreator.subscribe_all(queue, topics)
47
- log.info "Subscribed all topics to #{queue.name}"
48
+ logger.info "Subscribed all topics to #{queue.name}"
48
49
  true
49
50
  end
50
51
  end
@@ -52,7 +53,7 @@ module Circuitry
52
53
  def safe_aws(desc)
53
54
  yield
54
55
  rescue Aws::SQS::Errors::AccessDenied
55
- log.fatal("#{desc}: Access denied. Check your configured credentials.")
56
+ logger.fatal("#{desc}: Access denied. Check your configured credentials.")
56
57
  nil
57
58
  end
58
59
  end
@@ -30,10 +30,12 @@ module Circuitry
30
30
  raise ArgumentError, 'object cannot be nil' if object.nil?
31
31
  raise PublishError, 'AWS configuration is not set' unless can_publish?
32
32
 
33
+ message = object.to_json
34
+
33
35
  if async?
34
- process_asynchronously(& -> { publish_internal(topic_name, object) })
36
+ process_asynchronously { publish_internal(topic_name, message) }
35
37
  else
36
- publish_internal(topic_name, object)
38
+ publish_internal(topic_name, message)
37
39
  end
38
40
  end
39
41
 
@@ -43,14 +45,16 @@ module Circuitry
43
45
 
44
46
  protected
45
47
 
46
- def publish_internal(topic_name, object)
47
- # TODO: Don't use ruby timeout.
48
- # http://www.mikeperham.com/2015/05/08/timeout-rubys-most-dangerous-api/
49
- Timeout.timeout(timeout) do
50
- logger.info("Publishing message to #{topic_name}")
48
+ def publish_internal(topic_name, message)
49
+ middleware.invoke(topic_name, message) do
50
+ # TODO: Don't use ruby timeout.
51
+ # http://www.mikeperham.com/2015/05/08/timeout-rubys-most-dangerous-api/
52
+ Timeout.timeout(timeout) do
53
+ logger.info("Publishing message to #{topic_name}")
51
54
 
52
- topic = TopicCreator.find_or_create(topic_name)
53
- sns.publish(topic_arn: topic.arn, message: object.to_json)
55
+ topic = TopicCreator.find_or_create(topic_name)
56
+ sns.publish(topic_arn: topic.arn, message: message)
57
+ end
54
58
  end
55
59
  end
56
60
 
@@ -67,5 +71,9 @@ module Circuitry
67
71
  !value.nil? && !value.empty?
68
72
  end
69
73
  end
74
+
75
+ def middleware
76
+ Circuitry.config.publisher_middleware
77
+ end
70
78
  end
71
79
  end
@@ -119,7 +119,7 @@ module Circuitry
119
119
  end
120
120
 
121
121
  def process_messages_asynchronously(messages, &block)
122
- messages.each { |message| process_asynchronously(& -> { process_message(message, &block) }) }
122
+ messages.each { |message| process_asynchronously { process_message(message, &block) } }
123
123
  end
124
124
 
125
125
  def process_messages_synchronously(messages, &block)
@@ -139,21 +139,25 @@ module Circuitry
139
139
 
140
140
  def handle_message(message, &block)
141
141
  handled = try_with_lock(message.id) do
142
- begin
143
- # TODO: Don't use ruby timeout.
144
- # http://www.mikeperham.com/2015/05/08/timeout-rubys-most-dangerous-api/
145
- Timeout.timeout(timeout) do
146
- block.call(message.body, message.topic.name)
147
- end
148
- rescue => e
149
- logger.error("Error handling message #{message.id}: #{e}")
150
- raise e
142
+ middleware.invoke(message.topic.name, message.body) do
143
+ handle_message_with_timeout(message, &block)
151
144
  end
152
145
  end
153
146
 
154
147
  logger.info("Ignoring duplicate message #{message.id}") unless handled
155
148
  end
156
149
 
150
+ # TODO: Don't use ruby timeout.
151
+ # http://www.mikeperham.com/2015/05/08/timeout-rubys-most-dangerous-api/
152
+ def handle_message_with_timeout(message, &block)
153
+ Timeout.timeout(timeout) do
154
+ block.call(message.body, message.topic.name)
155
+ end
156
+ rescue => e
157
+ logger.error("Error handling message #{message.id}: #{e}")
158
+ raise e
159
+ end
160
+
157
161
  def try_with_lock(handle)
158
162
  if lock.soft_lock(handle)
159
163
  begin
@@ -188,5 +192,9 @@ module Circuitry
188
192
  !value.nil? && !value.empty?
189
193
  end
190
194
  end
195
+
196
+ def middleware
197
+ Circuitry.config.subscriber_middleware
198
+ end
191
199
  end
192
200
  end
@@ -1,3 +1,3 @@
1
1
  module Circuitry
2
- VERSION = '2.0.0'
2
+ VERSION = '2.1.0'
3
3
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: circuitry
3
3
  version: !ruby/object:Gem::Version
4
- version: 2.0.0
4
+ version: 2.1.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Matt Huggins
@@ -254,6 +254,8 @@ files:
254
254
  - lib/circuitry/locks/noop.rb
255
255
  - lib/circuitry/locks/redis.rb
256
256
  - lib/circuitry/message.rb
257
+ - lib/circuitry/middleware/chain.rb
258
+ - lib/circuitry/middleware/entry.rb
257
259
  - lib/circuitry/processor.rb
258
260
  - lib/circuitry/processors/batcher.rb
259
261
  - lib/circuitry/processors/forker.rb
@@ -291,7 +293,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
291
293
  version: '0'
292
294
  requirements: []
293
295
  rubyforge_project:
294
- rubygems_version: 2.5.1
296
+ rubygems_version: 2.4.8
295
297
  signing_key:
296
298
  specification_version: 4
297
299
  summary: Decouple ruby applications using Amazon SNS fanout with SQS processing.