eventq 2.0.0.rc1 → 2.0.0.rc2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +54 -24
- data/lib/eventq/aws.rb +1 -6
- data/lib/eventq/eventq_aws/aws_queue_worker.rb +17 -42
- data/lib/eventq/eventq_base/configuration.rb +2 -6
- data/lib/eventq/eventq_base/queue.rb +3 -0
- data/lib/eventq/eventq_base/serialization_providers.rb +2 -9
- data/lib/eventq/eventq_rabbitmq/rabbitmq_queue_client.rb +1 -6
- data/lib/eventq/eventq_rabbitmq/rabbitmq_queue_manager.rb +4 -12
- data/lib/eventq/eventq_rabbitmq/rabbitmq_queue_worker.rb +22 -56
- data/lib/eventq/queue_worker.rb +56 -1
- data/lib/eventq/rabbitmq.rb +2 -10
- metadata +19 -35
- data/lib/eventq/eventq_aws/README.md +0 -53
- data/lib/eventq/eventq_aws/jruby/aws_queue_worker.rb +0 -370
- data/lib/eventq/eventq_base/serialization_providers/jruby/oj/array_writer.rb +0 -20
- data/lib/eventq/eventq_base/serialization_providers/jruby/oj/attribute_writer.rb +0 -24
- data/lib/eventq/eventq_base/serialization_providers/jruby/oj/class_writer.rb +0 -20
- data/lib/eventq/eventq_base/serialization_providers/jruby/oj/date_time_writer.rb +0 -33
- data/lib/eventq/eventq_base/serialization_providers/jruby/oj/date_writer.rb +0 -22
- data/lib/eventq/eventq_base/serialization_providers/jruby/oj/hash_writer.rb +0 -18
- data/lib/eventq/eventq_base/serialization_providers/jruby/oj/rational_writer.rb +0 -20
- data/lib/eventq/eventq_base/serialization_providers/jruby/oj/serializer.rb +0 -17
- data/lib/eventq/eventq_base/serialization_providers/jruby/oj/time_writer.rb +0 -18
- data/lib/eventq/eventq_base/serialization_providers/jruby/oj/value_writer.rb +0 -16
- data/lib/eventq/eventq_base/serialization_providers/jruby/oj.rb +0 -10
- data/lib/eventq/eventq_base/serialization_providers/jruby/oj_serialization_provider.rb +0 -25
- data/lib/eventq/eventq_base/serialization_providers/jruby.rb +0 -2
- data/lib/eventq/eventq_rabbitmq/README.md +0 -36
- data/lib/eventq/eventq_rabbitmq/jruby/rabbitmq_queue_worker.rb +0 -367
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: f9337dac62d76513c21ca56ebfb129212d227a8979ef57e8bade4e3cafe445bf
|
4
|
+
data.tar.gz: 7b5e777c81e23f97500741a6a1ca27ee5e2e6a2d8d3134d4cf3544da5985c801
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 3eaa44ba2b8d2705ccd8bf358a115b50f3807d697f0d0455625b1107da5ff63e6a08e164bb20079936ba31a38072e047a72591f5a7d8e3446a009ec8d19d01f8
|
7
|
+
data.tar.gz: 7b98818050e8eb53eceb1f55d79e8a06ce08af15439e4d3fdbe66aab2de402525ddcf3990ba7687d76b410151b06a102f4a4edc8192491acd65810216843d32d
|
data/README.md
CHANGED
@@ -4,18 +4,16 @@
|
|
4
4
|
[![Maintainability](https://api.codeclimate.com/v1/badges/87205b497059e2733bdc/maintainability)](https://codeclimate.com/github/Sage/eventq/maintainability)
|
5
5
|
[![Test Coverage](https://api.codeclimate.com/v1/badges/87205b497059e2733bdc/test_coverage)](https://codeclimate.com/github/Sage/eventq/test_coverage)
|
6
6
|
|
7
|
-
Welcome to EventQ.
|
7
|
+
Welcome to EventQ.
|
8
8
|
|
9
9
|
EventQ is an event service bus framework for decoupling services and application processes.
|
10
10
|
|
11
11
|
Events are raised through the EventQ client and subscribers of the event types will be broadcast the event via a persistent queue for guaranteed delivery.
|
12
|
+
Existing solutions like ActiveJob work by assuming it posts directly to the queue provider. EventQ takes advantage of systems that fanout notifications.
|
13
|
+
This allows a notification to have multiple subscribers of which one is a message that EventQ can directly process.
|
12
14
|
|
13
15
|
EventQ has a base layer which allows different queue implementations to be created abstracting the specific queue implementation details away from your application code.
|
14
|
-
EventQ comes with two default
|
15
|
-
|
16
|
-
[AWS README](./lib/eventq/eventq_aws/README.md)
|
17
|
-
|
18
|
-
[RabbitMq README](./lib/eventq/eventq_rabbitmq/README.md)
|
16
|
+
EventQ comes with two default adapters, one for AWS SNS/SQS and another for RabbitMq (Fanout/Queue).
|
19
17
|
|
20
18
|
## Installation
|
21
19
|
|
@@ -56,14 +54,15 @@ A subscription queue should be defined to receive any events raised for the subs
|
|
56
54
|
**Attributes**
|
57
55
|
|
58
56
|
- **allow_retry** [Bool] [Optional] [Default=false] This determines if the queue should allow processing failures to be retried.
|
59
|
-
- **allow_retry_backoff** [Bool] [Optional] [Default=false] This is used to specify if failed messages that retry should incrementally backoff.
|
57
|
+
- **allow_retry_backoff** [Bool] [Optional] [Default=false] This is used to specify if failed messages that retry should incrementally backoff.
|
58
|
+
- **retry_back_off_grace** [Int] [Optional] [Default=0] This is the number of times to allow retries without applying retry back off if enabled.
|
60
59
|
- **dlq** [EventQ::Queue] [Optional] [Default=nil] A queue that will receive the messages which were not successfully processed after maximum number of receives by consumers. This is created at the same time as the parent queue.
|
61
60
|
- **max_retry_attempts** [Int] [Optional] [Default=5] This is used to specify the max number of times an event should be allowed to retry before failing.
|
62
61
|
- **max_receive_count** [Int] [Optional] [Default=30] The maximum number of times that a message can be received by consumers. When this value is exceeded for a message the message will be automatically sent to the Dead Letter Queue.
|
63
62
|
- **max_retry_delay** [Int] [Optional] This is used to specify the max retry delay that should apply when allowing incremental back off.
|
64
63
|
- **name** [String] [Required] This is the name of the queue, it must be unique.
|
65
64
|
- **require_signature** [Bool] [Optional] [Default=false] This is used to specify if messages within this queue must be signed.
|
66
|
-
- **retry_delay** [Int] [Optional] [Default=30000] This is used to specify the time delay in milliseconds before a failed message is re-added to the subscription queue.
|
65
|
+
- **retry_delay** [Int] [Optional] [Default=30000] This is used to specify the time delay in milliseconds before a failed message is re-added to the subscription queue.
|
67
66
|
|
68
67
|
**Example**
|
69
68
|
|
@@ -96,7 +95,7 @@ This method is called to subscribe a queue to an event type.
|
|
96
95
|
|
97
96
|
#create an instant of the queue definition
|
98
97
|
queue = DateChangeAddressQueue.new
|
99
|
-
|
98
|
+
|
100
99
|
#subscribe the queue definition to an event type
|
101
100
|
subscription_manager.subscribe('Data:Change:Address', queue)
|
102
101
|
|
@@ -112,7 +111,7 @@ This method is called to unsubscribe a queue.
|
|
112
111
|
|
113
112
|
#create an instant of the queue definition
|
114
113
|
queue = DateChangeAddressQueue.new
|
115
|
-
|
114
|
+
|
116
115
|
#subscribe the queue definition to an event type
|
117
116
|
subscription_manager.unsubscribe(queue)
|
118
117
|
|
@@ -132,7 +131,7 @@ The on_retry_exceeded method allows you to specify a block that should execute w
|
|
132
131
|
#Do something with the failed event
|
133
132
|
....
|
134
133
|
end
|
135
|
-
|
134
|
+
|
136
135
|
#### #on_retry
|
137
136
|
|
138
137
|
The on_retry method allows you to specify a block that should execute whenever an event fails to process and is retried. The event object passed to the block is a **[QueueMessage]** object, and the abort arg is a Boolean that specifies if the message was aborted (true or false).
|
@@ -147,7 +146,7 @@ The on_retry method allows you to specify a block that should execute whenever a
|
|
147
146
|
#Do something with the failed event
|
148
147
|
....
|
149
148
|
end
|
150
|
-
|
149
|
+
|
151
150
|
#### #on_error
|
152
151
|
|
153
152
|
The on_error method allows you to specify a block that should execute whenever an unhandled error occurs with the worker. The could be communication failures with the queue etc.
|
@@ -178,7 +177,7 @@ The start method is used to specify a block to process received events and start
|
|
178
177
|
> - **:sleep** [Number] [Optional] [Default=15] This is the number of seconds a thread should sleep before attempting to request another event from the subscription queue when no event has been received.
|
179
178
|
>
|
180
179
|
> - **:wait** [Bool] [Optional] This is used to specify that the start method should block the calling thread and wait until all parallel threads have finished. (This can be used to ensure that the background process running the worker does not exit).
|
181
|
-
>
|
180
|
+
>
|
182
181
|
> **Block arguments:**
|
183
182
|
> - **content** [Object] This is the content of the received event.
|
184
183
|
>
|
@@ -216,11 +215,11 @@ The **[QueueMessage]** is used internally to represent an event within the vario
|
|
216
215
|
- **content** [Object] This is the event content.
|
217
216
|
- **retry_attempts** [Int] This is the number of times this event message has been retried.
|
218
217
|
- **created** [DateTime] this is when the event was initial raised.
|
219
|
-
|
218
|
+
|
220
219
|
### Configuration
|
221
|
-
|
220
|
+
|
222
221
|
The `EventQ::Configuration` class allows global configuration options to be specified.
|
223
|
-
|
222
|
+
|
224
223
|
#### serialization_provider
|
225
224
|
|
226
225
|
This is used to specify the serialization provider that should be used for event serialization & deserialization.
|
@@ -236,7 +235,7 @@ This is used to specify the serialization provider that should be used for event
|
|
236
235
|
|
237
236
|
#set the serialization provider configuration to the JSON_PROVIDER
|
238
237
|
EventQ::Configuration.serialization_provider = EventQ::SerializationProviders::JSON_PROVIDER
|
239
|
-
|
238
|
+
|
240
239
|
#### signature_provider
|
241
240
|
|
242
241
|
This is used to specify the signature provider that should be used for message signing.
|
@@ -244,7 +243,7 @@ This is used to specify the signature provider that should be used for message s
|
|
244
243
|
> **Options:**
|
245
244
|
>
|
246
245
|
> - **SHA256** [Default] This is provider uses SHA256 to create message signatures.
|
247
|
-
|
246
|
+
|
248
247
|
#### signature_secret
|
249
248
|
|
250
249
|
This is used to specify the signature secret that should be used for message signing.
|
@@ -252,7 +251,7 @@ This is used to specify the signature secret that should be used for message sig
|
|
252
251
|
#set the signature secret
|
253
252
|
EventQ::Configuration.signature_secret = 'secret key'
|
254
253
|
|
255
|
-
|
254
|
+
|
256
255
|
### NonceManager
|
257
256
|
|
258
257
|
The NonceManager is used to prevent duplicate messages from being processed. Each event message that is raised is given a unique identifier, most message queue providers guarantee at least once delivery which may result in the message being delivered more than once. If your use case needs to enforce once only processing then
|
@@ -271,7 +270,7 @@ This method is called to configure the NonceManager, and must be called before s
|
|
271
270
|
**Example**
|
272
271
|
|
273
272
|
EventQ::NonceManager.configure(server: 'redis://127.0.0.1:6379')
|
274
|
-
|
273
|
+
|
275
274
|
|
276
275
|
### Namespace
|
277
276
|
|
@@ -292,14 +291,14 @@ This method is called to verify connection to a queue.
|
|
292
291
|
**Params:**
|
293
292
|
|
294
293
|
- **queue** [EventQ::Queue] [Required] This is a queue definition object.
|
295
|
-
|
294
|
+
|
296
295
|
**Return** [Boolean] (True or False)
|
297
296
|
|
298
297
|
|
299
298
|
**Example**
|
300
299
|
|
301
300
|
available = status_checker.queue?(queue)
|
302
|
-
|
301
|
+
|
303
302
|
####event_type?
|
304
303
|
|
305
304
|
This method is called to verify connection to an event_type (topic/exchange).
|
@@ -314,14 +313,45 @@ This method is called to verify connection to an event_type (topic/exchange).
|
|
314
313
|
**Example**
|
315
314
|
|
316
315
|
available = status_checker.event_type?(event_type)
|
317
|
-
|
318
316
|
|
317
|
+
## AWS Environment Variables
|
318
|
+
|
319
|
+
- **AWS_SQS_ENDPOINT** [String] [Optional] This is used to specify the endpoint of the SQS service to use.
|
320
|
+
- **AWS_SNS_ENDPOINT** [String] [Optional] This is used to specify the endpoint of the SNS service to use.
|
319
321
|
|
320
322
|
## Development
|
321
323
|
|
322
324
|
After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
|
323
325
|
|
324
|
-
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `
|
326
|
+
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in the file, `EVENTQ_VERSION`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
|
327
|
+
|
328
|
+
|
329
|
+
### Preparing the Docker images
|
330
|
+
|
331
|
+
Run the setup script of eventq to build the environment. This will create the `eventq` image.
|
332
|
+
|
333
|
+
$ cd script
|
334
|
+
$ ./setup.sh
|
335
|
+
|
336
|
+
### Running the tests
|
337
|
+
|
338
|
+
By default, the full test suite will run against the mock AWS services defined in the docker-compose.yml file. It also will run the tests for RabbitMq.
|
339
|
+
|
340
|
+
If you want to run the tests with AWS directly, you will need an AWS account. Put your credentials into the `.aws.env` file in the parent directory.
|
341
|
+
You will also need to comment out the AWS_* environment variables in the `docker-compose.yml` file
|
342
|
+
|
343
|
+
$ cp ../.aws.env.template ../.aws.env
|
344
|
+
$ vi ../.aws.env
|
345
|
+
|
346
|
+
Run the whole test suite:
|
347
|
+
|
348
|
+
$ cd script
|
349
|
+
$ ./test.sh
|
350
|
+
|
351
|
+
You can run the specs that don't depend on an AWS account with:
|
352
|
+
|
353
|
+
$ cd script
|
354
|
+
$ ./test.sh --tag ~integration
|
325
355
|
|
326
356
|
## Contributing
|
327
357
|
|
data/lib/eventq/aws.rb
CHANGED
@@ -8,12 +8,7 @@ require_relative './eventq_aws/aws_queue_client'
|
|
8
8
|
require_relative './eventq_aws/aws_queue_manager'
|
9
9
|
require_relative './eventq_aws/aws_subscription_manager'
|
10
10
|
require_relative './eventq_aws/aws_status_checker'
|
11
|
-
|
12
|
-
if RUBY_PLATFORM =~ /java/
|
13
|
-
require_relative './eventq_aws/jruby/aws_queue_worker'
|
14
|
-
else
|
15
|
-
require_relative './eventq_aws/aws_queue_worker'
|
16
|
-
end
|
11
|
+
require_relative './eventq_aws/aws_queue_worker'
|
17
12
|
|
18
13
|
module EventQ
|
19
14
|
def self.namespace
|
@@ -68,6 +68,11 @@ module EventQ
|
|
68
68
|
EventQ.logger.info("[#{self.class}] - Configuring. Queue Poll Wait: #{options[:queue_poll_wait]}")
|
69
69
|
end
|
70
70
|
|
71
|
+
# Logic for the RabbitMq adapter when a message is accepted
|
72
|
+
def acknowledge_message(poller, msg)
|
73
|
+
poller.delete_message(msg)
|
74
|
+
end
|
75
|
+
|
71
76
|
private
|
72
77
|
|
73
78
|
def process_message(msg, poller, queue, block)
|
@@ -77,51 +82,20 @@ module EventQ
|
|
77
82
|
payload = JSON.load(msg.body)
|
78
83
|
message = deserialize_message(payload[MESSAGE])
|
79
84
|
|
80
|
-
message_args = EventQ::MessageArgs.new(
|
81
|
-
type: message.type,
|
82
|
-
retry_attempts: retry_attempts,
|
83
|
-
context: message.context,
|
84
|
-
content_type: message.content_type,
|
85
|
-
id: message.id,
|
86
|
-
sent: message.created
|
87
|
-
)
|
88
|
-
|
89
|
-
EventQ.logger.info("[#{self.class}] - Message received. Retry Attempts: #{retry_attempts}")
|
90
|
-
|
91
85
|
@signature_provider_manager.validate_signature(message: message, queue: queue)
|
92
86
|
|
93
|
-
|
94
|
-
EventQ.logger.info("[#{self.class}] - Duplicate Message received. Ignoring message.")
|
95
|
-
return false
|
96
|
-
end
|
97
|
-
|
98
|
-
# begin worker block for queue message
|
99
|
-
begin
|
100
|
-
block.call(message.content, message_args)
|
87
|
+
status, message_args = context.process_message(block, message, retry_attempts, [poller, msg])
|
101
88
|
|
102
|
-
|
103
|
-
|
89
|
+
case status
|
90
|
+
when :duplicate
|
91
|
+
# don't do anything, this is previous logic. Not sure it is correct
|
92
|
+
when :accepted
|
93
|
+
# Acceptance was handled directly when QueueWorker#process_message was called
|
94
|
+
when :reject
|
95
|
+
reject_message(queue, poller, msg, retry_attempts, message, message_args.abort)
|
104
96
|
else
|
105
|
-
|
106
|
-
poller.delete_message(msg)
|
107
|
-
EventQ.logger.info("[#{self.class}] - Message acknowledged.")
|
108
|
-
end
|
109
|
-
|
110
|
-
rescue => e
|
111
|
-
EventQ.logger.error("[#{self.class}] - unhandled error while attempting to process a queue message")
|
112
|
-
EventQ.logger.error(e)
|
113
|
-
error = true
|
114
|
-
context.call_on_error_block(error: e, message: message)
|
115
|
-
end
|
116
|
-
|
117
|
-
if message_args.abort || error
|
118
|
-
EventQ::NonceManager.failed(message.id)
|
119
|
-
reject_message(queue, poller, msg, retry_attempts, message, message_args.abort)
|
120
|
-
else
|
121
|
-
EventQ::NonceManager.complete(message.id)
|
97
|
+
raise "Unrecognized status: #{status}"
|
122
98
|
end
|
123
|
-
|
124
|
-
true
|
125
99
|
end
|
126
100
|
|
127
101
|
def reject_message(queue, poller, msg, retry_attempts, message, abort)
|
@@ -140,15 +114,16 @@ module EventQ
|
|
140
114
|
|
141
115
|
EventQ.logger.info("[#{self.class}] - Message rejected requesting retry. Attempts: #{retry_attempts}")
|
142
116
|
|
117
|
+
retry_attempts = retry_attempts - queue.retry_back_off_grace
|
118
|
+
retry_attempts = 1 if retry_attempts < 1
|
119
|
+
|
143
120
|
if queue.allow_retry_back_off == true
|
144
|
-
EventQ.logger.debug { "[#{self.class}] - Calculating message back off retry delay. Attempts: #{retry_attempts} * Delay: #{queue.retry_delay}" }
|
145
121
|
visibility_timeout = (queue.retry_delay * retry_attempts) / 1000
|
146
122
|
if visibility_timeout > (queue.max_retry_delay / 1000)
|
147
123
|
EventQ.logger.debug { "[#{self.class}] - Max message back off retry delay reached." }
|
148
124
|
visibility_timeout = queue.max_retry_delay / 1000
|
149
125
|
end
|
150
126
|
else
|
151
|
-
EventQ.logger.debug { "[#{self.class}] - Setting fixed retry delay for message." }
|
152
127
|
visibility_timeout = queue.retry_delay / 1000
|
153
128
|
end
|
154
129
|
|
@@ -6,11 +6,7 @@ module EventQ
|
|
6
6
|
end
|
7
7
|
|
8
8
|
def self.serialization_provider
|
9
|
-
|
10
|
-
@serialization_provider ||= EventQ::SerializationProviders::JSON_PROVIDER
|
11
|
-
else
|
12
|
-
@serialization_provider ||= EventQ::SerializationProviders::OJ_PROVIDER
|
13
|
-
end
|
9
|
+
@serialization_provider ||= EventQ::SerializationProviders::OJ_PROVIDER
|
14
10
|
end
|
15
11
|
|
16
12
|
def self.signature_provider=(value)
|
@@ -30,4 +26,4 @@ module EventQ
|
|
30
26
|
end
|
31
27
|
|
32
28
|
end
|
33
|
-
end
|
29
|
+
end
|
@@ -9,6 +9,7 @@ module EventQ
|
|
9
9
|
attr_accessor :max_receive_count
|
10
10
|
attr_accessor :require_signature
|
11
11
|
attr_accessor :retry_delay
|
12
|
+
attr_accessor :retry_back_off_grace
|
12
13
|
|
13
14
|
def initialize
|
14
15
|
@allow_retry = false
|
@@ -22,6 +23,8 @@ module EventQ
|
|
22
23
|
@require_signature = false
|
23
24
|
# Default retry delay is 30 seconds
|
24
25
|
@retry_delay = 30000
|
26
|
+
# This is the amount of times to allow retry to occurr before back off is implemented
|
27
|
+
@retry_back_off_grace = 0
|
25
28
|
end
|
26
29
|
end
|
27
30
|
end
|
@@ -1,8 +1,5 @@
|
|
1
1
|
require_relative 'serialization_providers/json_serialization_provider'
|
2
|
-
|
3
|
-
require_relative 'serialization_providers/oj_serialization_provider'
|
4
|
-
end
|
5
|
-
require_relative 'serialization_providers/jruby'
|
2
|
+
require_relative 'serialization_providers/oj_serialization_provider'
|
6
3
|
require_relative 'serialization_providers/binary_serialization_provider'
|
7
4
|
|
8
5
|
module EventQ
|
@@ -15,11 +12,7 @@ module EventQ
|
|
15
12
|
class Manager
|
16
13
|
def initialize
|
17
14
|
@providers = {}
|
18
|
-
|
19
|
-
@providers[OJ_PROVIDER] = EventQ::SerializationProviders::JRuby::OjSerializationProvider
|
20
|
-
else
|
21
|
-
@providers[OJ_PROVIDER] = EventQ::SerializationProviders::OjSerializationProvider
|
22
|
-
end
|
15
|
+
@providers[OJ_PROVIDER] = EventQ::SerializationProviders::OjSerializationProvider
|
23
16
|
@providers[JSON_PROVIDER] = EventQ::SerializationProviders::JsonSerializationProvider
|
24
17
|
@providers[BINARY_PROVIDER] = EventQ::SerializationProviders::BinarySerializationProvider
|
25
18
|
end
|
@@ -39,12 +39,7 @@ module EventQ
|
|
39
39
|
end
|
40
40
|
|
41
41
|
def get_connection
|
42
|
-
|
43
|
-
conn = MarchHare.connect(connection_options)
|
44
|
-
else
|
45
|
-
conn = Bunny.new(connection_options)
|
46
|
-
end
|
47
|
-
|
42
|
+
conn = Bunny.new(connection_options)
|
48
43
|
conn.start
|
49
44
|
return conn
|
50
45
|
end
|
@@ -33,19 +33,11 @@ module EventQ
|
|
33
33
|
end
|
34
34
|
|
35
35
|
def pop_message(queue:)
|
36
|
-
|
37
|
-
|
38
|
-
|
39
|
-
return [nil,nil]
|
40
|
-
end
|
41
|
-
[headers.delivery_tag, payload]
|
42
|
-
else
|
43
|
-
headers, properties, payload = queue.pop({ :manual_ack => true, :block => true })
|
44
|
-
if headers == nil
|
45
|
-
return [nil,nil]
|
46
|
-
end
|
47
|
-
[headers.delivery_tag, payload]
|
36
|
+
headers, properties, payload = queue.pop({ :manual_ack => true, :block => true })
|
37
|
+
if headers == nil
|
38
|
+
return [nil,nil]
|
48
39
|
end
|
40
|
+
[headers.delivery_tag, payload]
|
49
41
|
end
|
50
42
|
|
51
43
|
def get_queue_exchange(channel, queue)
|
@@ -64,102 +64,68 @@ module EventQ
|
|
64
64
|
end
|
65
65
|
|
66
66
|
def reject_message(channel, message, delivery_tag, retry_exchange, queue, abort)
|
67
|
-
|
68
67
|
EventQ.logger.info("[#{self.class}] - Message rejected removing from queue.")
|
69
|
-
#reject the message to remove from queue
|
68
|
+
# reject the message to remove from queue
|
70
69
|
channel.reject(delivery_tag, false)
|
71
70
|
|
72
|
-
#check if the message retry limit has been exceeded
|
71
|
+
# check if the message retry limit has been exceeded
|
73
72
|
if message.retry_attempts >= queue.max_retry_attempts
|
74
|
-
|
75
73
|
EventQ.logger.info("[#{self.class}] - Message retry attempt limit exceeded. Msg: #{serialize_message(message)}")
|
76
74
|
|
77
75
|
context.call_on_retry_exceeded_block(message)
|
78
|
-
|
79
|
-
#check if the message is allowed to be retried
|
76
|
+
# check if the message is allowed to be retried
|
80
77
|
elsif queue.allow_retry
|
81
|
-
|
82
|
-
EventQ.logger.debug { "[#{self.class}] - Incrementing retry attempts count." }
|
83
78
|
message.retry_attempts += 1
|
79
|
+
retry_attempts = message.retry_attempts - queue.retry_back_off_grace
|
80
|
+
retry_attempts = 1 if retry_attempts < 1
|
84
81
|
|
85
82
|
if queue.allow_retry_back_off == true
|
86
|
-
|
87
|
-
|
88
|
-
if (message.retry_attempts * queue.retry_delay) > queue.max_retry_delay
|
83
|
+
message_ttl = retry_attempts * queue.retry_delay
|
84
|
+
if (retry_attempts * queue.retry_delay) > queue.max_retry_delay
|
89
85
|
EventQ.logger.debug { "[#{self.class}] - Max message back off retry delay reached." }
|
90
86
|
message_ttl = queue.max_retry_delay
|
91
87
|
end
|
92
88
|
else
|
93
|
-
EventQ.logger.debug { "[#{self.class}] - Setting fixed retry delay for message." }
|
94
89
|
message_ttl = queue.retry_delay
|
95
90
|
end
|
96
91
|
|
97
92
|
EventQ.logger.debug { "[#{self.class}] - Sending message for retry. Message TTL: #{message_ttl}" }
|
98
93
|
retry_exchange.publish(serialize_message(message), :expiration => message_ttl)
|
99
|
-
EventQ.logger.debug { "[#{self.class}] - Published message to retry exchange." }
|
100
94
|
|
101
95
|
context.call_on_retry_block(message)
|
102
|
-
|
103
96
|
end
|
104
97
|
|
105
98
|
return true
|
106
|
-
|
107
99
|
end
|
108
100
|
|
109
101
|
def configure(options = {})
|
110
102
|
options[:durable] ||= true
|
111
103
|
end
|
112
104
|
|
105
|
+
# Logic for the RabbitMq adapter when a message is accepted
|
106
|
+
def acknowledge_message(channel, delivery_tag)
|
107
|
+
channel.acknowledge(delivery_tag, false)
|
108
|
+
end
|
109
|
+
|
113
110
|
private
|
114
111
|
|
115
112
|
def process_message(payload, queue, channel, retry_exchange, delivery_tag, block)
|
116
|
-
abort = false
|
117
|
-
error = false
|
118
113
|
message = deserialize_message(payload)
|
119
|
-
|
120
|
-
EventQ.logger.info("[#{self.class}] - Message received. Retry Attempts: #{message.retry_attempts}")
|
114
|
+
retry_attempts = message.retry_attempts
|
121
115
|
|
122
116
|
@signature_provider_manager.validate_signature(message: message, queue: queue)
|
123
117
|
|
124
|
-
message_args =
|
125
|
-
type: message.type,
|
126
|
-
retry_attempts: message.retry_attempts,
|
127
|
-
context: message.context,
|
128
|
-
content_type: message.content_type,
|
129
|
-
id: message.id,
|
130
|
-
sent: message.created
|
131
|
-
)
|
132
|
-
|
133
|
-
if(!EventQ::NonceManager.is_allowed?(message.id))
|
134
|
-
EventQ.logger.info("[#{self.class}] - Duplicate Message received. Dropping message.")
|
135
|
-
channel.acknowledge(delivery_tag, false)
|
136
|
-
return false
|
137
|
-
end
|
138
|
-
|
139
|
-
# begin worker block for queue message
|
140
|
-
begin
|
141
|
-
block.call(message.content, message_args)
|
118
|
+
status, message_args = context.process_message(block, message, retry_attempts, [channel, delivery_tag])
|
142
119
|
|
143
|
-
|
144
|
-
|
145
|
-
EventQ.logger.info("[#{self.class}] - Message aborted.")
|
146
|
-
else
|
147
|
-
# accept the message as processed
|
120
|
+
case status
|
121
|
+
when :duplicate
|
148
122
|
channel.acknowledge(delivery_tag, false)
|
149
|
-
|
150
|
-
|
151
|
-
|
152
|
-
|
153
|
-
|
154
|
-
|
155
|
-
context.call_on_error_block(error: e, message: message)
|
156
|
-
end
|
157
|
-
|
158
|
-
if error || abort
|
159
|
-
EventQ::NonceManager.failed(message.id)
|
160
|
-
reject_message(channel, message, delivery_tag, retry_exchange, queue, abort)
|
161
|
-
else
|
162
|
-
EventQ::NonceManager.complete(message.id)
|
123
|
+
when :accepted
|
124
|
+
# Acceptance was handled directly when QueueWorker#process_message was called
|
125
|
+
when :reject
|
126
|
+
reject_message(channel, message, delivery_tag, retry_exchange, queue, message_args.abort)
|
127
|
+
else
|
128
|
+
raise "Unrecognized status: #{status}"
|
163
129
|
end
|
164
130
|
end
|
165
131
|
end
|
data/lib/eventq/queue_worker.rb
CHANGED
@@ -54,7 +54,7 @@ module EventQ
|
|
54
54
|
# So each fork gets its own copy of the @worker_status variable.
|
55
55
|
track_process(pid)
|
56
56
|
end
|
57
|
-
|
57
|
+
|
58
58
|
Process.waitall
|
59
59
|
else
|
60
60
|
# No need to track process/threads separately as we are in the main parent process,
|
@@ -108,6 +108,61 @@ module EventQ
|
|
108
108
|
raise Exceptions::WorkerThreadError, e.message, e.backtrace
|
109
109
|
end
|
110
110
|
|
111
|
+
# Method to be called by an adapter. This defines the common logic for processing a message.
|
112
|
+
# @param [Array] acceptance_args list of arguments that would be used to accept a message by an adapter.
|
113
|
+
# @return [Symbol, MessageArgs] :accepted, :duplicate, :reject
|
114
|
+
def process_message(block, message, retry_attempts, acceptance_args)
|
115
|
+
abort = false
|
116
|
+
error = false
|
117
|
+
status = nil
|
118
|
+
|
119
|
+
message_args = EventQ::MessageArgs.new(
|
120
|
+
type: message.type,
|
121
|
+
retry_attempts: retry_attempts,
|
122
|
+
context: message.context,
|
123
|
+
content_type: message.content_type,
|
124
|
+
id: message.id,
|
125
|
+
sent: message.created
|
126
|
+
)
|
127
|
+
|
128
|
+
EventQ.logger.info("[#{self.class}] - Message received. Retry Attempts: #{retry_attempts}")
|
129
|
+
|
130
|
+
if (!EventQ::NonceManager.is_allowed?(message.id))
|
131
|
+
EventQ.logger.info("[#{self.class}] - Duplicate Message received. Ignoring message.")
|
132
|
+
status = :duplicate
|
133
|
+
return status, message_args
|
134
|
+
end
|
135
|
+
|
136
|
+
# begin worker block for queue message
|
137
|
+
begin
|
138
|
+
block.call(message.content, message_args)
|
139
|
+
|
140
|
+
if message_args.abort == true
|
141
|
+
abort = true
|
142
|
+
EventQ.logger.info("[#{self.class}] - Message aborted.")
|
143
|
+
else
|
144
|
+
# accept the message as processed
|
145
|
+
status = :accepted
|
146
|
+
worker_adapter.acknowledge_message(*acceptance_args)
|
147
|
+
EventQ.logger.info("[#{self.class}] - Message acknowledged.")
|
148
|
+
end
|
149
|
+
rescue => e
|
150
|
+
EventQ.logger.error("[#{self.class}] - Unhandled error while attempting to process a queue message.")
|
151
|
+
EventQ.logger.error(e)
|
152
|
+
error = true
|
153
|
+
call_on_error_block(error: e, message: message)
|
154
|
+
end
|
155
|
+
|
156
|
+
if error || abort
|
157
|
+
EventQ::NonceManager.failed(message.id)
|
158
|
+
status = :reject
|
159
|
+
else
|
160
|
+
EventQ::NonceManager.complete(message.id)
|
161
|
+
end
|
162
|
+
|
163
|
+
[status, message_args]
|
164
|
+
end
|
165
|
+
|
111
166
|
def stop
|
112
167
|
EventQ.logger.info("[#{self.class}] - Stopping.")
|
113
168
|
@is_running = false
|
data/lib/eventq/rabbitmq.rb
CHANGED
@@ -1,20 +1,12 @@
|
|
1
1
|
# require 'eventq_base'
|
2
2
|
|
3
|
-
|
4
|
-
require 'march_hare'
|
5
|
-
else
|
6
|
-
require 'bunny'
|
7
|
-
end
|
3
|
+
require 'bunny'
|
8
4
|
|
9
5
|
require 'hash_kit'
|
10
6
|
require_relative './eventq_rabbitmq/rabbitmq_queue_client'
|
11
7
|
require_relative './eventq_rabbitmq/rabbitmq_queue_manager'
|
12
8
|
|
13
|
-
|
14
|
-
require_relative './eventq_rabbitmq/jruby/rabbitmq_queue_worker'
|
15
|
-
else
|
16
|
-
require_relative './eventq_rabbitmq/rabbitmq_queue_worker'
|
17
|
-
end
|
9
|
+
require_relative './eventq_rabbitmq/rabbitmq_queue_worker'
|
18
10
|
|
19
11
|
require_relative './eventq_rabbitmq/rabbitmq_subscription_manager'
|
20
12
|
require_relative './eventq_rabbitmq/rabbitmq_eventq_client'
|