circuitry 1.0.0 → 1.1.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 2399e4bc4f1e1d82e117415684b20c2920e34020
4
- data.tar.gz: 9b497dde60c50e60b945ef80be7dbe07382755e8
3
+ metadata.gz: e63a7c32a7c197c43ef244bdb5b3be5ae1544f39
4
+ data.tar.gz: f99bee8e6cc734e0ae334d31de01eb70851717e3
5
5
  SHA512:
6
- metadata.gz: b026a20b13d88fcfa09b7ab069578c236fca5dff3aa94f80b98e13aa4d2520b195a4c0cbed62f679211ee27cc54bb05be2f8657f6b738e63620422883b692be7
7
- data.tar.gz: 6248ef0005b4c98c9aa3d50cdf977a3426b78d8f8eac8a420ed4066ae258b2128e614e1ef091e5b829ba7eea1610546ec7b27692f554e2e2572f1fe5777149c8
6
+ metadata.gz: 4bb4da24b7d24c81b212c3c8f750d8bed4c85e8bde06f96e25f08c5d6291e1bed908948eec337caa193333e821cd02a01cfc94d5c86dcb3eeeddd24fab893da5
7
+ data.tar.gz: a54b2dc0d13eb5172e869163bef080e2a3095f09e3fa832ce6b79d0a1b508ec9fdf03610c4a98be83a3c7127b914ea82a398a935beb05d12f2e5891a3ed00933
data/README.md CHANGED
@@ -36,6 +36,8 @@ Circuitry.config do |c|
36
36
  HoneyBadger.notify(error)
37
37
  HoneyBadger.flush
38
38
  end
39
+ c.publish_async_strategy = :batch
40
+ c.subscribe_async_strategy = :thread
39
41
  end
40
42
  ```
41
43
 
@@ -52,12 +54,27 @@ Available configuration options include:
52
54
  * `error_handler`: An object that responds to `call` with two arguments: the
53
55
  deserialized message contents and the topic name used when publishing to SNS.
54
56
  *(optional, default: `nil`)*
57
+ * `publish_async_strategy`: One of `:fork`, `:thread`, or `:batch` that
58
+ determines how asynchronous publish requests are processed. *(optional,
59
+ default: `:fork`)*
60
+ * `:fork`: Forks a detached child process that immediately sends the request.
61
+ * `:thread`: Creates a new thread that immediately sends the request. Because
62
+ threads are not guaranteed to complete when the process exits, completion can
63
+ be ensured by calling `Circuitry.flush`.
64
+ * `:batch`: Stores the request in memory to be submitted later. Batched
65
+ requests must be manually sent by calling `Circuitry.flush`.
66
+ * `subscribe_async_strategy`: One of `:fork` or `:thread` that determines how
67
+ asynchronous subscribe requests are processed. *(optional, default: `:fork`)*
68
+ * `:fork`: Forks a detached child process that immediately begins querying the
69
+ queue.
70
+ * `:thread`: Creates a new thread that immediately sends begins querying the
71
+ queue.
55
72
 
56
73
  ### Publishing
57
74
 
58
75
  Publishing is done via the `Circuitry.publish` method. It accepts a topic name
59
- the represents the SNS topic along with any non-nil object, representing the data
60
- to be serialized. Whatever object is called will have its `to_json` method
76
+ that represents the SNS topic along with any non-nil object, representing the
77
+ data to be serialized. Whatever object is called will have its `to_json` method
61
78
  called for serialization.
62
79
 
63
80
  ```ruby
@@ -68,9 +85,11 @@ Circuitry.publish('any-topic-name', obj)
68
85
  The `publish` method also accepts options that impact instantiation of the
69
86
  `Publisher` object, which currently includes the following options.
70
87
 
71
- * `:async` - Whether or not publishing should occur in the background. Please
72
- refer to the [Asynchronous Support](#asynchronous-support) section for more
73
- details regarding this option. (default: false)
88
+ * `:async` - Whether or not publishing should occur in the background. Accepts
89
+ one of `:fork`, `:thread`, `:batch`, `true`, or `false`. Passing `true` uses
90
+ the `publish_async_strategy` value from the gem configuration. Please refer to
91
+ the [Asynchronous Support](#asynchronous-support) section for more details
92
+ regarding this option. *(default: `false`)*
74
93
 
75
94
  ```ruby
76
95
  obj = { foo: 'foo', bar: 'bar' }
@@ -89,8 +108,8 @@ publisher.publish('my-topic-name', obj)
89
108
  ### Subscribing
90
109
 
91
110
  Subscribing is done via the `Circuitry.subscribe` method. It accepts an SQS queue
92
- URL and takes a block for processing each message. This method performs
93
- synchronously by default, and as such does not return.
111
+ URL and takes a block for processing each message. This method **indefinitely
112
+ blocks**, processing messages as they are enqueued.
94
113
 
95
114
  ```ruby
96
115
  Circuitry.subscribe('https://sqs.REGION.amazonaws.com/ACCOUNT-ID/QUEUE-NAME') do |message, topic_name|
@@ -101,14 +120,18 @@ end
101
120
  The `subscribe` method also accepts options that impact instantiation of the
102
121
  `Subscriber` object, which currently includes the following options.
103
122
 
104
- * `:async` - Whether or not subscribing should occur in the background. Please
105
- refer to the [Asynchronous Support](#asynchronous-support) section for more
106
- details regarding this option. (default: false)
123
+ * `:async` - Whether or not subscribing should occur in the background. Accepts
124
+ one of `:fork`, `:thread`, `true`, or `false`. Passing `true` uses the
125
+ `subscribe_async_strategy` value from the gem configuration. Passing an
126
+ asynchronous value will cause messages to be handled concurrently, meaning
127
+ that queued messages *might not be processed in the order they're received*.
128
+ Please refer to the [Asynchronous Support](#asynchronous-support) section for
129
+ more details regarding this option. *(default: `false`)*
107
130
  * `:wait_time` - The number of seconds to wait for messages while connected to
108
131
  SQS. Anything above 0 results in long-polling, while 0 results in
109
- short-polling. (default: 10)
132
+ short-polling. *(default: 10)*
110
133
  * `:batch_size` - The number of messages to retrieve in a single SQS request.
111
- (default: 10)
134
+ *(default: 10)*
112
135
 
113
136
  ```ruby
114
137
  Circuitry.subscribe('https://...', async: true, wait_time: 60, batch_size: 20) do |message, topic_name|
@@ -129,17 +152,22 @@ end
129
152
 
130
153
  ### Asynchronous Support
131
154
 
132
- Publishing or subscribing asynchronously occurs by forking a child process. That
133
- child is then detached so that your application does not need to worry about
134
- waiting for the process to finish.
155
+ Publishing supports three asynchronous strategies (forking, threading, and
156
+ batching) while subscribing supports two (forking and threading).
157
+
158
+ #### Forking
159
+
160
+ When forking a child process, that child is detached so that your application
161
+ does not need to worry about waiting for the process to finish. Forked requests
162
+ begin processing immediately and do not have any overhead in terms of waiting for
163
+ them to complete.
135
164
 
136
165
  There are two important notes regarding forking in general as it relates to
137
166
  asynchronous support:
138
167
 
139
168
  1. Forking is not supported on all platforms (e.g.: Windows and NetBSD 4),
140
- requiring that your implementation use synchronous requests in such
141
- circumstances. You can determine if asynchronous requests will work by
142
- calling `Circuitry.platform_supports_async?`.
169
+ requiring that your implementation use synchronous requests or an alternative
170
+ asynchronous strategy in such circumstances.
143
171
 
144
172
  2. Forking results in resources being copied from the parent process to the child
145
173
  process. In order to prevent database connection errors and the like, you
@@ -164,6 +192,18 @@ asynchronous support:
164
192
  Refer to your adapter's documentation to determine how resources are handled
165
193
  with regards to forking.
166
194
 
195
+ #### Threading
196
+
197
+ Threaded publish and subscribe requests begin processing immediately. Unlike
198
+ forking, it's up to you to ensure that all threads complete before your
199
+ application exits. This can be done by calling `Circuitry.flush`.
200
+
201
+ #### Batching
202
+
203
+ Batched publish and subscribe requests are queued in memory and do not begin
204
+ processing until you explicit flush them. This can be done by calling
205
+ `Circuitry.flush`.
206
+
167
207
  ## Development
168
208
 
169
209
  After checking out the repo, run `bin/setup` to install dependencies. Then, run
data/circle.yml CHANGED
@@ -1,3 +1,7 @@
1
+ machine:
2
+ ruby:
3
+ version: 2.2.0
4
+
1
5
  dependencies:
2
6
  pre:
3
7
  - gem install bundler -v 1.8.9
@@ -1,17 +1,67 @@
1
+ require 'circuitry/processors/batcher'
2
+ require 'circuitry/processors/forker'
3
+ require 'circuitry/processors/threader'
4
+
1
5
  module Circuitry
2
6
  class NotSupportedError < StandardError; end
3
7
 
4
8
  module Concerns
5
9
  module Async
10
+ attr_reader :async
11
+
12
+ def self.included(base)
13
+ base.extend(ClassMethods)
14
+ end
15
+
16
+ module ClassMethods
17
+ def async_strategies
18
+ [:fork, :thread, :batch]
19
+ end
20
+
21
+ def default_async_strategy
22
+ raise NotImplementedError, "#{name} must implement class method `default_async_strategy`"
23
+ end
24
+ end
25
+
6
26
  def process_asynchronously(&block)
7
- raise NotSupportedError, 'Your platform does not support forking' unless platform_supports_async?
27
+ send(:"process_via_#{async}", &block)
28
+ end
29
+
30
+ def async=(value)
31
+ value = case value
32
+ when false, nil then false
33
+ when true then self.class.default_async_strategy
34
+ when *self.class.async_strategies then value
35
+ else raise ArgumentError, "Invalid value `#{value.inspect}`, must be one of #{[true, false].concat(self.class.async_strategies).inspect}"
36
+ end
37
+
38
+ if value == :fork && !platform_supports_forking?
39
+ raise NotSupportedError, 'Your platform does not support forking'
40
+ end
41
+
42
+ @async = value
43
+ end
44
+
45
+ def async?
46
+ !!async
47
+ end
48
+
49
+ private
50
+
51
+ def platform_supports_forking?
52
+ Process.respond_to?(:fork)
53
+ end
54
+
55
+ def process_via_fork(&block)
56
+ Processors::Forker.process(&block)
57
+ end
8
58
 
9
- pid = fork(&block)
10
- Process.detach(pid)
59
+ def process_via_thread(&block)
60
+ Processors::Threader.process(&block)
11
61
  end
12
62
 
13
- def platform_supports_async?
14
- Circuitry.platform_supports_async?
63
+ def process_via_batch(&block)
64
+ Processors::Batcher.process(&block)
15
65
  end
16
66
  end
17
67
  end
@@ -10,6 +10,18 @@ module Circuitry
10
10
  attribute :region, String, default: 'us-east-1'
11
11
  attribute :logger, Logger, default: Logger.new(STDERR)
12
12
  attribute :error_handler
13
+ attribute :publish_async_strategy, Symbol, default: ->(page, attribute) { :fork }
14
+ attribute :subscribe_async_strategy, Symbol, default: ->(page, attribute) { :fork }
15
+
16
+ def publish_async_strategy=(value)
17
+ validate(value, Publisher.async_strategies)
18
+ super
19
+ end
20
+
21
+ def subscribe_async_strategy=(value)
22
+ validate(value, Subscriber.async_strategies)
23
+ super
24
+ end
13
25
 
14
26
  def aws_options
15
27
  {
@@ -18,5 +30,13 @@ module Circuitry
18
30
  region: region,
19
31
  }
20
32
  end
33
+
34
+ private
35
+
36
+ def validate(value, permitted_values)
37
+ unless permitted_values.include?(value)
38
+ raise ArgumentError, "invalid value `#{value}`, must be one of #{permitted_values.inspect}"
39
+ end
40
+ end
21
41
  end
22
42
  end
@@ -0,0 +1,36 @@
1
+ module Circuitry
2
+ module Processor
3
+ def process
4
+ raise NotImplementedError, "#{self} must implement class method `process`"
5
+ end
6
+
7
+ def flush
8
+ raise NotImplementedError, "#{self} must implement class method `flush`"
9
+ end
10
+
11
+ protected
12
+
13
+ def safely_process(&block)
14
+ begin
15
+ block.call
16
+ rescue => e
17
+ logger.error("Error handling message: #{e}")
18
+ error_handler.call(e) if error_handler
19
+ end
20
+ end
21
+
22
+ def pool
23
+ @pool ||= []
24
+ end
25
+
26
+ private
27
+
28
+ def logger
29
+ Circuitry.config.logger
30
+ end
31
+
32
+ def error_handler
33
+ Circuitry.config.error_handler
34
+ end
35
+ end
36
+ end
@@ -0,0 +1,22 @@
1
+ require 'circuitry/processor'
2
+
3
+ module Circuitry
4
+ module Processors
5
+ module Batcher
6
+ class << self
7
+ include Processor
8
+
9
+ def process(&block)
10
+ raise ArgumentError, 'no block given' unless block_given?
11
+ pool << block
12
+ end
13
+
14
+ def flush
15
+ pool.each { |block| safely_process(&block) }
16
+ ensure
17
+ pool.clear
18
+ end
19
+ end
20
+ end
21
+ end
22
+ end
@@ -0,0 +1,19 @@
1
+ require 'circuitry/processor'
2
+
3
+ module Circuitry
4
+ module Processors
5
+ module Forker
6
+ class << self
7
+ include Processor
8
+
9
+ def process(&block)
10
+ pid = fork { safely_process(&block) }
11
+ Process.detach(pid)
12
+ end
13
+
14
+ def flush
15
+ end
16
+ end
17
+ end
18
+ end
19
+ end
@@ -0,0 +1,22 @@
1
+ require 'circuitry/processor'
2
+
3
+ module Circuitry
4
+ module Processors
5
+ module Threader
6
+ class << self
7
+ include Processor
8
+
9
+ def process(&block)
10
+ raise ArgumentError, 'no block given' unless block_given?
11
+ pool << Thread.new { safely_process(&block) }
12
+ end
13
+
14
+ def flush
15
+ pool.each(&:join)
16
+ ensure
17
+ pool.clear
18
+ end
19
+ end
20
+ end
21
+ end
22
+ end
@@ -17,7 +17,7 @@ module Circuitry
17
17
  def initialize(options = {})
18
18
  options = DEFAULT_OPTIONS.merge(options)
19
19
 
20
- @async = !!options[:async]
20
+ self.async = options[:async]
21
21
  end
22
22
 
23
23
  def publish(topic_name, object)
@@ -41,8 +41,8 @@ module Circuitry
41
41
  end
42
42
  end
43
43
 
44
- def async?
45
- @async
44
+ def self.default_async_strategy
45
+ Circuitry.config.publish_async_strategy
46
46
  end
47
47
 
48
48
  private
@@ -26,10 +26,10 @@ module Circuitry
26
26
 
27
27
  options = DEFAULT_OPTIONS.merge(options)
28
28
 
29
- @queue = queue
30
- @async = !!options[:async]
31
- @wait_time = options[:wait_time]
32
- @batch_size = options[:batch_size]
29
+ self.queue = queue
30
+ self.async = options[:async]
31
+ self.wait_time = options[:wait_time]
32
+ self.batch_size = options[:batch_size]
33
33
  end
34
34
 
35
35
  def subscribe(&block)
@@ -40,28 +40,28 @@ module Circuitry
40
40
  return
41
41
  end
42
42
 
43
- process = -> do
44
- loop do
45
- begin
46
- receive_messages(&block)
47
- rescue *CONNECTION_ERRORS => e
48
- logger.error "Connection error to #{queue}: #{e}"
49
- raise SubscribeError.new(e)
50
- end
43
+ loop do
44
+ begin
45
+ receive_messages(&block)
46
+ rescue *CONNECTION_ERRORS => e
47
+ logger.error("Connection error to #{queue}: #{e}")
48
+ raise SubscribeError.new(e)
51
49
  end
52
50
  end
51
+ end
53
52
 
54
- if async?
55
- process_asynchronously(&process)
56
- else
57
- process.call
58
- end
53
+ def self.async_strategies
54
+ super - [:batch]
59
55
  end
60
56
 
61
- def async?
62
- @async
57
+ def self.default_async_strategy
58
+ Circuitry.config.subscribe_async_strategy
63
59
  end
64
60
 
61
+ protected
62
+
63
+ attr_writer :queue, :wait_time, :batch_size
64
+
65
65
  private
66
66
 
67
67
  def receive_messages(&block)
@@ -70,7 +70,15 @@ module Circuitry
70
70
  return if messages.empty?
71
71
 
72
72
  messages.each do |message|
73
- process_message(message, &block)
73
+ process = -> do
74
+ process_message(message, &block)
75
+ end
76
+
77
+ if async?
78
+ process_asynchronously(&process)
79
+ else
80
+ process.call
81
+ end
74
82
  end
75
83
  end
76
84
 
@@ -78,12 +86,12 @@ module Circuitry
78
86
  message = Message.new(message)
79
87
 
80
88
  unless message.nil?
81
- logger.info "Processing message #{message.id}"
89
+ logger.info("Processing message #{message.id}")
82
90
  handle_message(message, &block)
83
91
  delete_message(message)
84
92
  end
85
93
  rescue => e
86
- logger.error "Error processing message #{message.id}: #{e}"
94
+ logger.error("Error processing message #{message.id}: #{e}")
87
95
  error_handler.call(e) if error_handler
88
96
  end
89
97
 
@@ -1,3 +1,3 @@
1
1
  module Circuitry
2
- VERSION = '1.0.0'
2
+ VERSION = '1.1.0'
3
3
  end
data/lib/circuitry.rb CHANGED
@@ -1,4 +1,8 @@
1
1
  require 'circuitry/version'
2
+ require 'circuitry/processor'
3
+ require 'circuitry/processors/batcher'
4
+ require 'circuitry/processors/forker'
5
+ require 'circuitry/processors/threader'
2
6
  require 'circuitry/configuration'
3
7
  require 'circuitry/publisher'
4
8
  require 'circuitry/subscriber'
@@ -18,7 +22,9 @@ module Circuitry
18
22
  Subscriber.new(queue, options).subscribe(&block)
19
23
  end
20
24
 
21
- def self.platform_supports_async?
22
- Process.respond_to?(:fork)
25
+ def self.flush
26
+ Processors.constants.each do |const|
27
+ Processors.const_get(const).flush
28
+ end
23
29
  end
24
30
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: circuitry
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.0
4
+ version: 1.1.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Matt Huggins
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2015-06-25 00:00:00.000000000 Z
11
+ date: 2015-07-08 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: fog-aws
@@ -143,6 +143,10 @@ files:
143
143
  - lib/circuitry/concerns/async.rb
144
144
  - lib/circuitry/configuration.rb
145
145
  - lib/circuitry/message.rb
146
+ - lib/circuitry/processor.rb
147
+ - lib/circuitry/processors/batcher.rb
148
+ - lib/circuitry/processors/forker.rb
149
+ - lib/circuitry/processors/threader.rb
146
150
  - lib/circuitry/publisher.rb
147
151
  - lib/circuitry/services/sns.rb
148
152
  - lib/circuitry/services/sqs.rb