circuitry 2.0.0 → 2.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +9 -5
- data/README.md +90 -6
- data/lib/circuitry.rb +1 -0
- data/lib/circuitry/configuration.rb +12 -0
- data/lib/circuitry/middleware/chain.rb +82 -0
- data/lib/circuitry/middleware/entry.rb +20 -0
- data/lib/circuitry/provisioner.rb +9 -8
- data/lib/circuitry/publisher.rb +17 -9
- data/lib/circuitry/subscriber.rb +18 -10
- data/lib/circuitry/version.rb +1 -1
- metadata +4 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 14368085a767ad616196eb22569c3c981af716d0
|
4
|
+
data.tar.gz: 639f43f5b3e192241e5b142c5251331ee8f86362
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 15d2f7452c6f1324850b6b83a825ecc9c5bc7895319660b785917625dbf0a991f92ce7f4303c239df4eb520b8c128e427fd69c85069e687986fdc198d693266b
|
7
|
+
data.tar.gz: 8b4073842d2dca9813ee809ffcbab89ce8949ba148edbfa7d32531be5a79764eccc4f067489decbccccd143f86afa7db8767c55b34b9be64337ca9bc18aa8b41
|
data/CHANGELOG.md
CHANGED
@@ -1,9 +1,13 @@
|
|
1
|
-
## Circuitry 2.
|
1
|
+
## Circuitry 2.1.0 (Jan 28, 2016)
|
2
2
|
|
3
|
-
* Added
|
4
|
-
|
5
|
-
|
6
|
-
|
3
|
+
* Added publisher and subscriber middleware. *Matt Huggins*
|
4
|
+
|
5
|
+
## Circuitry 2.0.0 (Jan 28, 2016)
|
6
|
+
|
7
|
+
* Added subscriber_queue_name config. *Brandon Croft*
|
8
|
+
* Added publisher_topic_names config. *Brandon Croft*
|
9
|
+
* Added CLI and rake provisioning of queues and topics as defined by config. *Brandon Croft*
|
10
|
+
* Removed the requirement to provide a SQS URL to the subscriber. *Brandon Croft*
|
7
11
|
|
8
12
|
## Circuitry 1.4.1 (Jan 21, 2016)
|
9
13
|
|
data/README.md
CHANGED
@@ -87,18 +87,26 @@ Available configuration options include:
|
|
87
87
|
managing shared resources such as database connections that require closing,
|
88
88
|
It is only called when implementing the `:fork` async strategy. *(optional,
|
89
89
|
default: `nil`)*
|
90
|
+
* `publisher_topic_names`: An array of topic names that your publishing application will
|
91
|
+
publish on. This configuration is only used during provisioning via `rake circuitry:setup`
|
90
92
|
* `subscriber_queue_name`: The name of the SQS queue that your subscriber application
|
91
93
|
will listen to. This queue will be created or configured during `rake circuitry:setup`
|
92
94
|
*(optional, default: `nil`)*
|
93
95
|
* `subscriber_dead_letter_queue_name`: The name of the SQS dead letter queue that will be
|
94
96
|
used after all retries fail. This queue will be created and configured during `rake
|
95
97
|
circuitry:setup` *(optional, default: `<subscriber_queue_name>-failures`)*
|
96
|
-
* `
|
97
|
-
|
98
|
+
* `publisher_middleware`: A chain of middleware that sent messages must go through.
|
99
|
+
Please refer to the [Middleware](#middleware) section for more details regarding this
|
100
|
+
option.
|
101
|
+
* `subscriber_middleware`: A chain of middleware that received messages must go through.
|
102
|
+
Please refer to the [Middleware](#middleware) section for more details regarding this
|
103
|
+
option.
|
98
104
|
|
99
105
|
### Provisioning
|
100
106
|
|
101
|
-
You can automatically provision SQS queues, SNS topics, and the subscriptions between them using
|
107
|
+
You can automatically provision SQS queues, SNS topics, and the subscriptions between them using
|
108
|
+
two methods: the circuitry CLI or the `rake circuitry:setup` task. The rake task will provision the
|
109
|
+
subscriber queue and publishing topics that are configured within your application.
|
102
110
|
|
103
111
|
```ruby
|
104
112
|
Circuitry.config do |c|
|
@@ -107,7 +115,9 @@ Circuitry.config do |c|
|
|
107
115
|
end
|
108
116
|
```
|
109
117
|
|
110
|
-
When provisioning, a dead letter queue is also created using the name "<queue_name>-failures" and a
|
118
|
+
When provisioning, a dead letter queue is also created using the name "<queue_name>-failures" and a
|
119
|
+
redrive policy of 8 retries to that dead letter queue is configured. You can customize the dead
|
120
|
+
letter queue name in your configuration.
|
111
121
|
|
112
122
|
Run `ruby bin/circuitry help provision` for help using CLI provisioning.
|
113
123
|
|
@@ -152,8 +162,8 @@ publisher.publish('my-topic-name', obj)
|
|
152
162
|
|
153
163
|
### Subscribing
|
154
164
|
|
155
|
-
Subscribing is done via the `Circuitry.subscribe` method. It accepts a block for processing each
|
156
|
-
blocks**, processing messages as they are enqueued.
|
165
|
+
Subscribing is done via the `Circuitry.subscribe` method. It accepts a block for processing each
|
166
|
+
message. This method **indefinitely blocks**, processing messages as they are enqueued.
|
157
167
|
|
158
168
|
```ruby
|
159
169
|
Circuitry.subscribe do |message, topic_name|
|
@@ -422,6 +432,80 @@ connection = PG.connect(...)
|
|
422
432
|
Circuitry.config.lock_strategy = DatabaseLockStrategy.new(connection: connection)
|
423
433
|
```
|
424
434
|
|
435
|
+
### Middleware
|
436
|
+
|
437
|
+
Circuitry middleware can be used to perform additional processing around a message
|
438
|
+
being sent by a publisher or received by a subscriber. Some examples of processing
|
439
|
+
that belong here are monitoring or encryption specific to your application.
|
440
|
+
|
441
|
+
Middleware can be added to the publisher, the subscriber, or both. A middleware
|
442
|
+
class is defined by an (optional) `#initialize` method that accepts any number of
|
443
|
+
arguments, as well as a `#call` method that accepts the `topic` string, `message`
|
444
|
+
string, and a block for continuing processing.
|
445
|
+
|
446
|
+
For example, a simple logging middleware might look something like the following:
|
447
|
+
|
448
|
+
```ruby
|
449
|
+
class LoggerMiddleware
|
450
|
+
attr_reader :namespace, :logger
|
451
|
+
|
452
|
+
def initialize(namespace:, logger: Logger.new(STDOUT))
|
453
|
+
self.namespace = namespace
|
454
|
+
self.logger = logger
|
455
|
+
end
|
456
|
+
|
457
|
+
def call(topic, message)
|
458
|
+
logger.info("#{namespace} (start): #{topic} - #{message}")
|
459
|
+
yield
|
460
|
+
ensure
|
461
|
+
logger.info("#{namespace} (done): #{topic} - #{message}")
|
462
|
+
end
|
463
|
+
|
464
|
+
private
|
465
|
+
|
466
|
+
attr_writer :namespace, :logger
|
467
|
+
end
|
468
|
+
```
|
469
|
+
|
470
|
+
Adding the middleware to the stack happens through the Circuitry config.
|
471
|
+
|
472
|
+
```ruby
|
473
|
+
Circuitry.config do |config|
|
474
|
+
# single-line format
|
475
|
+
circuitry.publisher_middleware.add LoggerMiddleware, namespace: 'publisher'
|
476
|
+
circuitry.subscriber_middleware.add LoggerMiddleware, namespace: 'subscriber', logger: Rails.logger
|
477
|
+
|
478
|
+
# block format
|
479
|
+
circuitry.publisher_middleware do |chain|
|
480
|
+
chain.add LoggerMiddleware, namespace: 'publisher'
|
481
|
+
end
|
482
|
+
|
483
|
+
circuitry.subscriber_middleware do |chain|
|
484
|
+
chain.add LoggerMiddleware, namespace: 'subscriber', logger: Rails.logger
|
485
|
+
end
|
486
|
+
end
|
487
|
+
```
|
488
|
+
|
489
|
+
Both `publisher_middleware` and `subscriber_middleware` respond to a handful of methods that can be
|
490
|
+
used for configuring your middleware:
|
491
|
+
|
492
|
+
* `#add`: Appends a middleware class to the end of the chain. If the class already exists, it is
|
493
|
+
replaced.
|
494
|
+
* `middleware.add NewMiddleware, arg1, arg2, ...`
|
495
|
+
* `#prepend`: Prepends a middleware class to the beginning of the chain. If the class already
|
496
|
+
exists, it is replaced.
|
497
|
+
* `middleware.prepend NewMiddleware, arg1, arg2, ...`
|
498
|
+
* `#remove`: Removes a middleware class from anywhere in the chain.
|
499
|
+
* `middleware.remove NewMiddleware`
|
500
|
+
* `#insert_before`: Injects a middleware class before another middleware class in the chain. If
|
501
|
+
the other class does not exist in the chain, this behaves the same as `#prepend`.
|
502
|
+
* `middleware.insert_before ExistingMiddleware, NewMiddleware, arg1, arg2...`
|
503
|
+
* `#insert_after`: Injects a middleware class after another middleware class in the chain. If the
|
504
|
+
other class does not exist in the chain, this behaves the same as `#add`.
|
505
|
+
* `middleware.insert_after ExistingMiddleware, NewMiddleware, arg1, arg2...`
|
506
|
+
* `#clear`: Removes all middleware classes from the chain.
|
507
|
+
* `middleware.clear`
|
508
|
+
|
425
509
|
## Development
|
426
510
|
|
427
511
|
After checking out the repo, run `bin/setup` to install dependencies. Then, run
|
data/lib/circuitry.rb
CHANGED
@@ -5,6 +5,7 @@ require 'circuitry/locks/memcache'
|
|
5
5
|
require 'circuitry/locks/memory'
|
6
6
|
require 'circuitry/locks/noop'
|
7
7
|
require 'circuitry/locks/redis'
|
8
|
+
require 'circuitry/middleware/chain'
|
8
9
|
require 'circuitry/processor'
|
9
10
|
require 'circuitry/processors/batcher'
|
10
11
|
require 'circuitry/processors/forker'
|
@@ -34,6 +34,18 @@ module Circuitry
|
|
34
34
|
super || "#{subscriber_queue_name}-failures"
|
35
35
|
end
|
36
36
|
|
37
|
+
def subscriber_middleware
|
38
|
+
@subscriber_middleware ||= Circuitry::Middleware::Chain.new
|
39
|
+
yield @subscriber_middleware if block_given?
|
40
|
+
@subscriber_middleware
|
41
|
+
end
|
42
|
+
|
43
|
+
def publisher_middleware
|
44
|
+
@publisher_middleware ||= Circuitry::Middleware::Chain.new
|
45
|
+
yield @publisher_middleware if block_given?
|
46
|
+
@publisher_middleware
|
47
|
+
end
|
48
|
+
|
37
49
|
def aws_options
|
38
50
|
{
|
39
51
|
access_key_id: access_key,
|
@@ -0,0 +1,82 @@
|
|
1
|
+
require 'circuitry/middleware/entry'
|
2
|
+
|
3
|
+
module Circuitry
|
4
|
+
module Middleware
|
5
|
+
class Chain
|
6
|
+
include Enumerable
|
7
|
+
|
8
|
+
def each(&block)
|
9
|
+
entries.each(&block)
|
10
|
+
end
|
11
|
+
|
12
|
+
def entries
|
13
|
+
@entries ||= []
|
14
|
+
end
|
15
|
+
|
16
|
+
def add(klass, *args)
|
17
|
+
remove(klass) if exists?(klass)
|
18
|
+
entries << Entry.new(klass, *args)
|
19
|
+
end
|
20
|
+
|
21
|
+
def remove(klass)
|
22
|
+
entries.delete_if { |entry| entry.klass == klass }
|
23
|
+
end
|
24
|
+
|
25
|
+
def prepend(klass, *args)
|
26
|
+
remove(klass) if exists?(klass)
|
27
|
+
entries.unshift(Entry.new(klass, *args))
|
28
|
+
end
|
29
|
+
|
30
|
+
def insert_before(old_klass, new_klass, *args)
|
31
|
+
new_entry = build_or_replace_entry(new_klass, *args)
|
32
|
+
i = entries.index { |entry| entry.klass == old_klass } || 0
|
33
|
+
entries.insert(i, new_entry)
|
34
|
+
end
|
35
|
+
|
36
|
+
def insert_after(old_klass, new_klass, *args)
|
37
|
+
new_entry = build_or_replace_entry(new_klass, *args)
|
38
|
+
i = entries.index { |entry| entry.klass == old_klass } || entries.size - 1
|
39
|
+
entries.insert(i + 1, new_entry)
|
40
|
+
end
|
41
|
+
|
42
|
+
def exists?(klass)
|
43
|
+
any? { |entry| entry.klass == klass }
|
44
|
+
end
|
45
|
+
|
46
|
+
def build
|
47
|
+
map(&:build)
|
48
|
+
end
|
49
|
+
|
50
|
+
def clear
|
51
|
+
entries.clear
|
52
|
+
end
|
53
|
+
|
54
|
+
def invoke(*args)
|
55
|
+
chain = build.dup
|
56
|
+
|
57
|
+
traverse_chain = -> do
|
58
|
+
if chain.empty?
|
59
|
+
yield
|
60
|
+
else
|
61
|
+
chain.shift.call(*args, &traverse_chain)
|
62
|
+
end
|
63
|
+
end
|
64
|
+
|
65
|
+
traverse_chain.call
|
66
|
+
end
|
67
|
+
|
68
|
+
private
|
69
|
+
|
70
|
+
def build_or_replace_entry(klass, *args)
|
71
|
+
i = entries.index { |entry| entry.klass == klass }
|
72
|
+
entry = i.nil? ? Entry.new(klass, *args) : entries.delete_at(i)
|
73
|
+
|
74
|
+
if entry.args == args
|
75
|
+
entry
|
76
|
+
else
|
77
|
+
Entry.new(klass, *args)
|
78
|
+
end
|
79
|
+
end
|
80
|
+
end
|
81
|
+
end
|
82
|
+
end
|
@@ -0,0 +1,20 @@
|
|
1
|
+
module Circuitry
|
2
|
+
module Middleware
|
3
|
+
class Entry
|
4
|
+
attr_reader :klass, :args
|
5
|
+
|
6
|
+
def initialize(klass, *args)
|
7
|
+
self.klass = klass
|
8
|
+
self.args = args
|
9
|
+
end
|
10
|
+
|
11
|
+
def build
|
12
|
+
klass.new(*args)
|
13
|
+
end
|
14
|
+
|
15
|
+
private
|
16
|
+
|
17
|
+
attr_writer :klass, :args
|
18
|
+
end
|
19
|
+
end
|
20
|
+
end
|
@@ -4,12 +4,11 @@ require 'circuitry/subscription_creator'
|
|
4
4
|
|
5
5
|
module Circuitry
|
6
6
|
class Provisioner
|
7
|
-
attr_reader :
|
8
|
-
attr_reader :config
|
7
|
+
attr_reader :config, :logger
|
9
8
|
|
10
9
|
def initialize(config, logger: Logger.new(STDOUT))
|
11
|
-
|
12
|
-
|
10
|
+
self.config = config
|
11
|
+
self.logger = logger
|
13
12
|
end
|
14
13
|
|
15
14
|
def run
|
@@ -20,13 +19,15 @@ module Circuitry
|
|
20
19
|
|
21
20
|
private
|
22
21
|
|
22
|
+
attr_writer :config, :logger
|
23
|
+
|
23
24
|
def create_queue
|
24
25
|
safe_aws('Create Queue') do
|
25
26
|
queue = Circuitry::QueueCreator.find_or_create(
|
26
27
|
config.subscriber_queue_name,
|
27
28
|
dead_letter_queue_name: config.subscriber_dead_letter_queue_name
|
28
29
|
)
|
29
|
-
|
30
|
+
logger.info "Created queue #{queue.url}"
|
30
31
|
queue
|
31
32
|
end
|
32
33
|
end
|
@@ -35,7 +36,7 @@ module Circuitry
|
|
35
36
|
safe_aws('Create Topics') do
|
36
37
|
config.publisher_topic_names.map do |topic_name|
|
37
38
|
topic = Circuitry::TopicCreator.find_or_create(topic_name)
|
38
|
-
|
39
|
+
logger.info "Created topic #{topic.name}"
|
39
40
|
topic
|
40
41
|
end
|
41
42
|
end
|
@@ -44,7 +45,7 @@ module Circuitry
|
|
44
45
|
def subscribe_topics(queue, topics)
|
45
46
|
safe_aws('Subscribe Topics') do
|
46
47
|
Circuitry::SubscriptionCreator.subscribe_all(queue, topics)
|
47
|
-
|
48
|
+
logger.info "Subscribed all topics to #{queue.name}"
|
48
49
|
true
|
49
50
|
end
|
50
51
|
end
|
@@ -52,7 +53,7 @@ module Circuitry
|
|
52
53
|
def safe_aws(desc)
|
53
54
|
yield
|
54
55
|
rescue Aws::SQS::Errors::AccessDenied
|
55
|
-
|
56
|
+
logger.fatal("#{desc}: Access denied. Check your configured credentials.")
|
56
57
|
nil
|
57
58
|
end
|
58
59
|
end
|
data/lib/circuitry/publisher.rb
CHANGED
@@ -30,10 +30,12 @@ module Circuitry
|
|
30
30
|
raise ArgumentError, 'object cannot be nil' if object.nil?
|
31
31
|
raise PublishError, 'AWS configuration is not set' unless can_publish?
|
32
32
|
|
33
|
+
message = object.to_json
|
34
|
+
|
33
35
|
if async?
|
34
|
-
process_asynchronously
|
36
|
+
process_asynchronously { publish_internal(topic_name, message) }
|
35
37
|
else
|
36
|
-
publish_internal(topic_name,
|
38
|
+
publish_internal(topic_name, message)
|
37
39
|
end
|
38
40
|
end
|
39
41
|
|
@@ -43,14 +45,16 @@ module Circuitry
|
|
43
45
|
|
44
46
|
protected
|
45
47
|
|
46
|
-
def publish_internal(topic_name,
|
47
|
-
|
48
|
-
|
49
|
-
|
50
|
-
|
48
|
+
def publish_internal(topic_name, message)
|
49
|
+
middleware.invoke(topic_name, message) do
|
50
|
+
# TODO: Don't use ruby timeout.
|
51
|
+
# http://www.mikeperham.com/2015/05/08/timeout-rubys-most-dangerous-api/
|
52
|
+
Timeout.timeout(timeout) do
|
53
|
+
logger.info("Publishing message to #{topic_name}")
|
51
54
|
|
52
|
-
|
53
|
-
|
55
|
+
topic = TopicCreator.find_or_create(topic_name)
|
56
|
+
sns.publish(topic_arn: topic.arn, message: message)
|
57
|
+
end
|
54
58
|
end
|
55
59
|
end
|
56
60
|
|
@@ -67,5 +71,9 @@ module Circuitry
|
|
67
71
|
!value.nil? && !value.empty?
|
68
72
|
end
|
69
73
|
end
|
74
|
+
|
75
|
+
def middleware
|
76
|
+
Circuitry.config.publisher_middleware
|
77
|
+
end
|
70
78
|
end
|
71
79
|
end
|
data/lib/circuitry/subscriber.rb
CHANGED
@@ -119,7 +119,7 @@ module Circuitry
|
|
119
119
|
end
|
120
120
|
|
121
121
|
def process_messages_asynchronously(messages, &block)
|
122
|
-
messages.each { |message| process_asynchronously
|
122
|
+
messages.each { |message| process_asynchronously { process_message(message, &block) } }
|
123
123
|
end
|
124
124
|
|
125
125
|
def process_messages_synchronously(messages, &block)
|
@@ -139,21 +139,25 @@ module Circuitry
|
|
139
139
|
|
140
140
|
def handle_message(message, &block)
|
141
141
|
handled = try_with_lock(message.id) do
|
142
|
-
|
143
|
-
|
144
|
-
# http://www.mikeperham.com/2015/05/08/timeout-rubys-most-dangerous-api/
|
145
|
-
Timeout.timeout(timeout) do
|
146
|
-
block.call(message.body, message.topic.name)
|
147
|
-
end
|
148
|
-
rescue => e
|
149
|
-
logger.error("Error handling message #{message.id}: #{e}")
|
150
|
-
raise e
|
142
|
+
middleware.invoke(message.topic.name, message.body) do
|
143
|
+
handle_message_with_timeout(message, &block)
|
151
144
|
end
|
152
145
|
end
|
153
146
|
|
154
147
|
logger.info("Ignoring duplicate message #{message.id}") unless handled
|
155
148
|
end
|
156
149
|
|
150
|
+
# TODO: Don't use ruby timeout.
|
151
|
+
# http://www.mikeperham.com/2015/05/08/timeout-rubys-most-dangerous-api/
|
152
|
+
def handle_message_with_timeout(message, &block)
|
153
|
+
Timeout.timeout(timeout) do
|
154
|
+
block.call(message.body, message.topic.name)
|
155
|
+
end
|
156
|
+
rescue => e
|
157
|
+
logger.error("Error handling message #{message.id}: #{e}")
|
158
|
+
raise e
|
159
|
+
end
|
160
|
+
|
157
161
|
def try_with_lock(handle)
|
158
162
|
if lock.soft_lock(handle)
|
159
163
|
begin
|
@@ -188,5 +192,9 @@ module Circuitry
|
|
188
192
|
!value.nil? && !value.empty?
|
189
193
|
end
|
190
194
|
end
|
195
|
+
|
196
|
+
def middleware
|
197
|
+
Circuitry.config.subscriber_middleware
|
198
|
+
end
|
191
199
|
end
|
192
200
|
end
|
data/lib/circuitry/version.rb
CHANGED
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: circuitry
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 2.
|
4
|
+
version: 2.1.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Matt Huggins
|
@@ -254,6 +254,8 @@ files:
|
|
254
254
|
- lib/circuitry/locks/noop.rb
|
255
255
|
- lib/circuitry/locks/redis.rb
|
256
256
|
- lib/circuitry/message.rb
|
257
|
+
- lib/circuitry/middleware/chain.rb
|
258
|
+
- lib/circuitry/middleware/entry.rb
|
257
259
|
- lib/circuitry/processor.rb
|
258
260
|
- lib/circuitry/processors/batcher.rb
|
259
261
|
- lib/circuitry/processors/forker.rb
|
@@ -291,7 +293,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
291
293
|
version: '0'
|
292
294
|
requirements: []
|
293
295
|
rubyforge_project:
|
294
|
-
rubygems_version: 2.
|
296
|
+
rubygems_version: 2.4.8
|
295
297
|
signing_key:
|
296
298
|
specification_version: 4
|
297
299
|
summary: Decouple ruby applications using Amazon SNS fanout with SQS processing.
|