eventq 2.0.0.rc1 → 2.0.0.rc2

Sign up to get free protection for your applications and to get access to all the features.
Files changed (30) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +54 -24
  3. data/lib/eventq/aws.rb +1 -6
  4. data/lib/eventq/eventq_aws/aws_queue_worker.rb +17 -42
  5. data/lib/eventq/eventq_base/configuration.rb +2 -6
  6. data/lib/eventq/eventq_base/queue.rb +3 -0
  7. data/lib/eventq/eventq_base/serialization_providers.rb +2 -9
  8. data/lib/eventq/eventq_rabbitmq/rabbitmq_queue_client.rb +1 -6
  9. data/lib/eventq/eventq_rabbitmq/rabbitmq_queue_manager.rb +4 -12
  10. data/lib/eventq/eventq_rabbitmq/rabbitmq_queue_worker.rb +22 -56
  11. data/lib/eventq/queue_worker.rb +56 -1
  12. data/lib/eventq/rabbitmq.rb +2 -10
  13. metadata +19 -35
  14. data/lib/eventq/eventq_aws/README.md +0 -53
  15. data/lib/eventq/eventq_aws/jruby/aws_queue_worker.rb +0 -370
  16. data/lib/eventq/eventq_base/serialization_providers/jruby/oj/array_writer.rb +0 -20
  17. data/lib/eventq/eventq_base/serialization_providers/jruby/oj/attribute_writer.rb +0 -24
  18. data/lib/eventq/eventq_base/serialization_providers/jruby/oj/class_writer.rb +0 -20
  19. data/lib/eventq/eventq_base/serialization_providers/jruby/oj/date_time_writer.rb +0 -33
  20. data/lib/eventq/eventq_base/serialization_providers/jruby/oj/date_writer.rb +0 -22
  21. data/lib/eventq/eventq_base/serialization_providers/jruby/oj/hash_writer.rb +0 -18
  22. data/lib/eventq/eventq_base/serialization_providers/jruby/oj/rational_writer.rb +0 -20
  23. data/lib/eventq/eventq_base/serialization_providers/jruby/oj/serializer.rb +0 -17
  24. data/lib/eventq/eventq_base/serialization_providers/jruby/oj/time_writer.rb +0 -18
  25. data/lib/eventq/eventq_base/serialization_providers/jruby/oj/value_writer.rb +0 -16
  26. data/lib/eventq/eventq_base/serialization_providers/jruby/oj.rb +0 -10
  27. data/lib/eventq/eventq_base/serialization_providers/jruby/oj_serialization_provider.rb +0 -25
  28. data/lib/eventq/eventq_base/serialization_providers/jruby.rb +0 -2
  29. data/lib/eventq/eventq_rabbitmq/README.md +0 -36
  30. data/lib/eventq/eventq_rabbitmq/jruby/rabbitmq_queue_worker.rb +0 -367
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 29d2e0dbde90c017daf4729cf39dd998c499b09f5a54d329d0546048f812b729
4
- data.tar.gz: 08ab2c25998abdaf811169d26b1140107c671d1040091cc2294d0b5c6df87baf
3
+ metadata.gz: f9337dac62d76513c21ca56ebfb129212d227a8979ef57e8bade4e3cafe445bf
4
+ data.tar.gz: 7b5e777c81e23f97500741a6a1ca27ee5e2e6a2d8d3134d4cf3544da5985c801
5
5
  SHA512:
6
- metadata.gz: 82af850fe3d231f5966150a655286e057fcb71bea23900e6924381d186dcadf594011c9054eb7a9c87e35fb0cb768dca8dc166de81bd54801f94016d431e9bdd
7
- data.tar.gz: 13d4ee3700ee280297cd731c25c9448c00171384c53b21bc35d69cc6c23c9c3b275a0c79d327620d58ce7051134a42237125c0a02f0bd4238fa73d5d4c9f02b0
6
+ metadata.gz: 3eaa44ba2b8d2705ccd8bf358a115b50f3807d697f0d0455625b1107da5ff63e6a08e164bb20079936ba31a38072e047a72591f5a7d8e3446a009ec8d19d01f8
7
+ data.tar.gz: 7b98818050e8eb53eceb1f55d79e8a06ce08af15439e4d3fdbe66aab2de402525ddcf3990ba7687d76b410151b06a102f4a4edc8192491acd65810216843d32d
data/README.md CHANGED
@@ -4,18 +4,16 @@
4
4
  [![Maintainability](https://api.codeclimate.com/v1/badges/87205b497059e2733bdc/maintainability)](https://codeclimate.com/github/Sage/eventq/maintainability)
5
5
  [![Test Coverage](https://api.codeclimate.com/v1/badges/87205b497059e2733bdc/test_coverage)](https://codeclimate.com/github/Sage/eventq/test_coverage)
6
6
 
7
- Welcome to EventQ.
7
+ Welcome to EventQ.
8
8
 
9
9
  EventQ is an event service bus framework for decoupling services and application processes.
10
10
 
11
11
  Events are raised through the EventQ client and subscribers of the event types will be broadcast the event via a persistent queue for guaranteed delivery.
12
+ Existing solutions like ActiveJob work by assuming it posts directly to the queue provider. EventQ takes advantage of systems that fanout notifications.
13
+ This allows a notification to have multiple subscribers of which one is a message that EventQ can directly process.
12
14
 
13
15
  EventQ has a base layer which allows different queue implementations to be created abstracting the specific queue implementation details away from your application code.
14
- EventQ comes with two default adapter that support RabbitMq and AWS SNS/SQS
15
-
16
- [AWS README](./lib/eventq/eventq_aws/README.md)
17
-
18
- [RabbitMq README](./lib/eventq/eventq_rabbitmq/README.md)
16
+ EventQ comes with two default adapters, one for AWS SNS/SQS and another for RabbitMq (Fanout/Queue).
19
17
 
20
18
  ## Installation
21
19
 
@@ -56,14 +54,15 @@ A subscription queue should be defined to receive any events raised for the subs
56
54
  **Attributes**
57
55
 
58
56
  - **allow_retry** [Bool] [Optional] [Default=false] This determines if the queue should allow processing failures to be retried.
59
- - **allow_retry_backoff** [Bool] [Optional] [Default=false] This is used to specify if failed messages that retry should incrementally backoff.
57
+ - **allow_retry_backoff** [Bool] [Optional] [Default=false] This is used to specify if failed messages that retry should incrementally backoff.
58
+ - **retry_back_off_grace** [Int] [Optional] [Default=0] This is the number of times to allow retries without applying retry back off if enabled.
60
59
  - **dlq** [EventQ::Queue] [Optional] [Default=nil] A queue that will receive the messages which were not successfully processed after maximum number of receives by consumers. This is created at the same time as the parent queue.
61
60
  - **max_retry_attempts** [Int] [Optional] [Default=5] This is used to specify the max number of times an event should be allowed to retry before failing.
62
61
  - **max_receive_count** [Int] [Optional] [Default=30] The maximum number of times that a message can be received by consumers. When this value is exceeded for a message the message will be automatically sent to the Dead Letter Queue.
63
62
  - **max_retry_delay** [Int] [Optional] This is used to specify the max retry delay that should apply when allowing incremental back off.
64
63
  - **name** [String] [Required] This is the name of the queue, it must be unique.
65
64
  - **require_signature** [Bool] [Optional] [Default=false] This is used to specify if messages within this queue must be signed.
66
- - **retry_delay** [Int] [Optional] [Default=30000] This is used to specify the time delay in milliseconds before a failed message is re-added to the subscription queue.
65
+ - **retry_delay** [Int] [Optional] [Default=30000] This is used to specify the time delay in milliseconds before a failed message is re-added to the subscription queue.
67
66
 
68
67
  **Example**
69
68
 
@@ -96,7 +95,7 @@ This method is called to subscribe a queue to an event type.
96
95
 
97
96
  #create an instant of the queue definition
98
97
  queue = DateChangeAddressQueue.new
99
-
98
+
100
99
  #subscribe the queue definition to an event type
101
100
  subscription_manager.subscribe('Data:Change:Address', queue)
102
101
 
@@ -112,7 +111,7 @@ This method is called to unsubscribe a queue.
112
111
 
113
112
  #create an instant of the queue definition
114
113
  queue = DateChangeAddressQueue.new
115
-
114
+
116
115
  #subscribe the queue definition to an event type
117
116
  subscription_manager.unsubscribe(queue)
118
117
 
@@ -132,7 +131,7 @@ The on_retry_exceeded method allows you to specify a block that should execute w
132
131
  #Do something with the failed event
133
132
  ....
134
133
  end
135
-
134
+
136
135
  #### #on_retry
137
136
 
138
137
  The on_retry method allows you to specify a block that should execute whenever an event fails to process and is retried. The event object passed to the block is a **[QueueMessage]** object, and the abort arg is a Boolean that specifies if the message was aborted (true or false).
@@ -147,7 +146,7 @@ The on_retry method allows you to specify a block that should execute whenever a
147
146
  #Do something with the failed event
148
147
  ....
149
148
  end
150
-
149
+
151
150
  #### #on_error
152
151
 
153
152
  The on_error method allows you to specify a block that should execute whenever an unhandled error occurs with the worker. The could be communication failures with the queue etc.
@@ -178,7 +177,7 @@ The start method is used to specify a block to process received events and start
178
177
  > - **:sleep** [Number] [Optional] [Default=15] This is the number of seconds a thread should sleep before attempting to request another event from the subscription queue when no event has been received.
179
178
  >
180
179
  > - **:wait** [Bool] [Optional] This is used to specify that the start method should block the calling thread and wait until all parallel threads have finished. (This can be used to ensure that the background process running the worker does not exit).
181
- >
180
+ >
182
181
  > **Block arguments:**
183
182
  > - **content** [Object] This is the content of the received event.
184
183
  >
@@ -216,11 +215,11 @@ The **[QueueMessage]** is used internally to represent an event within the vario
216
215
  - **content** [Object] This is the event content.
217
216
  - **retry_attempts** [Int] This is the number of times this event message has been retried.
218
217
  - **created** [DateTime] this is when the event was initial raised.
219
-
218
+
220
219
  ### Configuration
221
-
220
+
222
221
  The `EventQ::Configuration` class allows global configuration options to be specified.
223
-
222
+
224
223
  #### serialization_provider
225
224
 
226
225
  This is used to specify the serialization provider that should be used for event serialization & deserialization.
@@ -236,7 +235,7 @@ This is used to specify the serialization provider that should be used for event
236
235
 
237
236
  #set the serialization provider configuration to the JSON_PROVIDER
238
237
  EventQ::Configuration.serialization_provider = EventQ::SerializationProviders::JSON_PROVIDER
239
-
238
+
240
239
  #### signature_provider
241
240
 
242
241
  This is used to specify the signature provider that should be used for message signing.
@@ -244,7 +243,7 @@ This is used to specify the signature provider that should be used for message s
244
243
  > **Options:**
245
244
  >
246
245
  > - **SHA256** [Default] This is provider uses SHA256 to create message signatures.
247
-
246
+
248
247
  #### signature_secret
249
248
 
250
249
  This is used to specify the signature secret that should be used for message signing.
@@ -252,7 +251,7 @@ This is used to specify the signature secret that should be used for message sig
252
251
  #set the signature secret
253
252
  EventQ::Configuration.signature_secret = 'secret key'
254
253
 
255
-
254
+
256
255
  ### NonceManager
257
256
 
258
257
  The NonceManager is used to prevent duplicate messages from being processed. Each event message that is raised is given a unique identifier, most message queue providers guarantee at least once delivery which may result in the message being delivered more than once. If your use case needs to enforce once only processing then
@@ -271,7 +270,7 @@ This method is called to configure the NonceManager, and must be called before s
271
270
  **Example**
272
271
 
273
272
  EventQ::NonceManager.configure(server: 'redis://127.0.0.1:6379')
274
-
273
+
275
274
 
276
275
  ### Namespace
277
276
 
@@ -292,14 +291,14 @@ This method is called to verify connection to a queue.
292
291
  **Params:**
293
292
 
294
293
  - **queue** [EventQ::Queue] [Required] This is a queue definition object.
295
-
294
+
296
295
  **Return** [Boolean] (True or False)
297
296
 
298
297
 
299
298
  **Example**
300
299
 
301
300
  available = status_checker.queue?(queue)
302
-
301
+
303
302
  ####event_type?
304
303
 
305
304
  This method is called to verify connection to an event_type (topic/exchange).
@@ -314,14 +313,45 @@ This method is called to verify connection to an event_type (topic/exchange).
314
313
  **Example**
315
314
 
316
315
  available = status_checker.event_type?(event_type)
317
-
318
316
 
317
+ ## AWS Environment Variables
318
+
319
+ - **AWS_SQS_ENDPOINT** [String] [Optional] This is used to specify the endpoint of the SQS service to use.
320
+ - **AWS_SNS_ENDPOINT** [String] [Optional] This is used to specify the endpoint of the SNS service to use.
319
321
 
320
322
  ## Development
321
323
 
322
324
  After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
323
325
 
324
- To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
326
+ To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in the file, `EVENTQ_VERSION`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
327
+
328
+
329
+ ### Preparing the Docker images
330
+
331
+ Run the setup script of eventq to build the environment. This will create the `eventq` image.
332
+
333
+ $ cd script
334
+ $ ./setup.sh
335
+
336
+ ### Running the tests
337
+
338
+ By default, the full test suite will run against the mock AWS services defined in the docker-compose.yml file. It also will run the tests for RabbitMq.
339
+
340
+ If you want to run the tests with AWS directly, you will need an AWS account. Put your credentials into the `.aws.env` file in the parent directory.
341
+ You will also need to comment out the AWS_* environment variables in the `docker-compose.yml` file
342
+
343
+ $ cp ../.aws.env.template ../.aws.env
344
+ $ vi ../.aws.env
345
+
346
+ Run the whole test suite:
347
+
348
+ $ cd script
349
+ $ ./test.sh
350
+
351
+ You can run the specs that don't depend on an AWS account with:
352
+
353
+ $ cd script
354
+ $ ./test.sh --tag ~integration
325
355
 
326
356
  ## Contributing
327
357
 
data/lib/eventq/aws.rb CHANGED
@@ -8,12 +8,7 @@ require_relative './eventq_aws/aws_queue_client'
8
8
  require_relative './eventq_aws/aws_queue_manager'
9
9
  require_relative './eventq_aws/aws_subscription_manager'
10
10
  require_relative './eventq_aws/aws_status_checker'
11
-
12
- if RUBY_PLATFORM =~ /java/
13
- require_relative './eventq_aws/jruby/aws_queue_worker'
14
- else
15
- require_relative './eventq_aws/aws_queue_worker'
16
- end
11
+ require_relative './eventq_aws/aws_queue_worker'
17
12
 
18
13
  module EventQ
19
14
  def self.namespace
@@ -68,6 +68,11 @@ module EventQ
68
68
  EventQ.logger.info("[#{self.class}] - Configuring. Queue Poll Wait: #{options[:queue_poll_wait]}")
69
69
  end
70
70
 
71
+ # Logic for the RabbitMq adapter when a message is accepted
72
+ def acknowledge_message(poller, msg)
73
+ poller.delete_message(msg)
74
+ end
75
+
71
76
  private
72
77
 
73
78
  def process_message(msg, poller, queue, block)
@@ -77,51 +82,20 @@ module EventQ
77
82
  payload = JSON.load(msg.body)
78
83
  message = deserialize_message(payload[MESSAGE])
79
84
 
80
- message_args = EventQ::MessageArgs.new(
81
- type: message.type,
82
- retry_attempts: retry_attempts,
83
- context: message.context,
84
- content_type: message.content_type,
85
- id: message.id,
86
- sent: message.created
87
- )
88
-
89
- EventQ.logger.info("[#{self.class}] - Message received. Retry Attempts: #{retry_attempts}")
90
-
91
85
  @signature_provider_manager.validate_signature(message: message, queue: queue)
92
86
 
93
- if(!EventQ::NonceManager.is_allowed?(message.id))
94
- EventQ.logger.info("[#{self.class}] - Duplicate Message received. Ignoring message.")
95
- return false
96
- end
97
-
98
- # begin worker block for queue message
99
- begin
100
- block.call(message.content, message_args)
87
+ status, message_args = context.process_message(block, message, retry_attempts, [poller, msg])
101
88
 
102
- if message_args.abort == true
103
- EventQ.logger.info("[#{self.class}] - Message aborted.")
89
+ case status
90
+ when :duplicate
91
+ # don't do anything, this is previous logic. Not sure it is correct
92
+ when :accepted
93
+ # Acceptance was handled directly when QueueWorker#process_message was called
94
+ when :reject
95
+ reject_message(queue, poller, msg, retry_attempts, message, message_args.abort)
104
96
  else
105
- # accept the message as processed
106
- poller.delete_message(msg)
107
- EventQ.logger.info("[#{self.class}] - Message acknowledged.")
108
- end
109
-
110
- rescue => e
111
- EventQ.logger.error("[#{self.class}] - unhandled error while attempting to process a queue message")
112
- EventQ.logger.error(e)
113
- error = true
114
- context.call_on_error_block(error: e, message: message)
115
- end
116
-
117
- if message_args.abort || error
118
- EventQ::NonceManager.failed(message.id)
119
- reject_message(queue, poller, msg, retry_attempts, message, message_args.abort)
120
- else
121
- EventQ::NonceManager.complete(message.id)
97
+ raise "Unrecognized status: #{status}"
122
98
  end
123
-
124
- true
125
99
  end
126
100
 
127
101
  def reject_message(queue, poller, msg, retry_attempts, message, abort)
@@ -140,15 +114,16 @@ module EventQ
140
114
 
141
115
  EventQ.logger.info("[#{self.class}] - Message rejected requesting retry. Attempts: #{retry_attempts}")
142
116
 
117
+ retry_attempts = retry_attempts - queue.retry_back_off_grace
118
+ retry_attempts = 1 if retry_attempts < 1
119
+
143
120
  if queue.allow_retry_back_off == true
144
- EventQ.logger.debug { "[#{self.class}] - Calculating message back off retry delay. Attempts: #{retry_attempts} * Delay: #{queue.retry_delay}" }
145
121
  visibility_timeout = (queue.retry_delay * retry_attempts) / 1000
146
122
  if visibility_timeout > (queue.max_retry_delay / 1000)
147
123
  EventQ.logger.debug { "[#{self.class}] - Max message back off retry delay reached." }
148
124
  visibility_timeout = queue.max_retry_delay / 1000
149
125
  end
150
126
  else
151
- EventQ.logger.debug { "[#{self.class}] - Setting fixed retry delay for message." }
152
127
  visibility_timeout = queue.retry_delay / 1000
153
128
  end
154
129
 
@@ -6,11 +6,7 @@ module EventQ
6
6
  end
7
7
 
8
8
  def self.serialization_provider
9
- if RUBY_PLATFORM =~ /java/
10
- @serialization_provider ||= EventQ::SerializationProviders::JSON_PROVIDER
11
- else
12
- @serialization_provider ||= EventQ::SerializationProviders::OJ_PROVIDER
13
- end
9
+ @serialization_provider ||= EventQ::SerializationProviders::OJ_PROVIDER
14
10
  end
15
11
 
16
12
  def self.signature_provider=(value)
@@ -30,4 +26,4 @@ module EventQ
30
26
  end
31
27
 
32
28
  end
33
- end
29
+ end
@@ -9,6 +9,7 @@ module EventQ
9
9
  attr_accessor :max_receive_count
10
10
  attr_accessor :require_signature
11
11
  attr_accessor :retry_delay
12
+ attr_accessor :retry_back_off_grace
12
13
 
13
14
  def initialize
14
15
  @allow_retry = false
@@ -22,6 +23,8 @@ module EventQ
22
23
  @require_signature = false
23
24
  # Default retry delay is 30 seconds
24
25
  @retry_delay = 30000
26
+ # This is the amount of times to allow retry to occurr before back off is implemented
27
+ @retry_back_off_grace = 0
25
28
  end
26
29
  end
27
30
  end
@@ -1,8 +1,5 @@
1
1
  require_relative 'serialization_providers/json_serialization_provider'
2
- unless RUBY_PLATFORM =~ /java/
3
- require_relative 'serialization_providers/oj_serialization_provider'
4
- end
5
- require_relative 'serialization_providers/jruby'
2
+ require_relative 'serialization_providers/oj_serialization_provider'
6
3
  require_relative 'serialization_providers/binary_serialization_provider'
7
4
 
8
5
  module EventQ
@@ -15,11 +12,7 @@ module EventQ
15
12
  class Manager
16
13
  def initialize
17
14
  @providers = {}
18
- if RUBY_PLATFORM =~ /java/
19
- @providers[OJ_PROVIDER] = EventQ::SerializationProviders::JRuby::OjSerializationProvider
20
- else
21
- @providers[OJ_PROVIDER] = EventQ::SerializationProviders::OjSerializationProvider
22
- end
15
+ @providers[OJ_PROVIDER] = EventQ::SerializationProviders::OjSerializationProvider
23
16
  @providers[JSON_PROVIDER] = EventQ::SerializationProviders::JsonSerializationProvider
24
17
  @providers[BINARY_PROVIDER] = EventQ::SerializationProviders::BinarySerializationProvider
25
18
  end
@@ -39,12 +39,7 @@ module EventQ
39
39
  end
40
40
 
41
41
  def get_connection
42
- if RUBY_PLATFORM =~ /java/
43
- conn = MarchHare.connect(connection_options)
44
- else
45
- conn = Bunny.new(connection_options)
46
- end
47
-
42
+ conn = Bunny.new(connection_options)
48
43
  conn.start
49
44
  return conn
50
45
  end
@@ -33,19 +33,11 @@ module EventQ
33
33
  end
34
34
 
35
35
  def pop_message(queue:)
36
- if RUBY_PLATFORM =~ /java/
37
- headers, payload = queue.pop({ :ack => true, :block => true })
38
- if headers == nil
39
- return [nil,nil]
40
- end
41
- [headers.delivery_tag, payload]
42
- else
43
- headers, properties, payload = queue.pop({ :manual_ack => true, :block => true })
44
- if headers == nil
45
- return [nil,nil]
46
- end
47
- [headers.delivery_tag, payload]
36
+ headers, properties, payload = queue.pop({ :manual_ack => true, :block => true })
37
+ if headers == nil
38
+ return [nil,nil]
48
39
  end
40
+ [headers.delivery_tag, payload]
49
41
  end
50
42
 
51
43
  def get_queue_exchange(channel, queue)
@@ -64,102 +64,68 @@ module EventQ
64
64
  end
65
65
 
66
66
  def reject_message(channel, message, delivery_tag, retry_exchange, queue, abort)
67
-
68
67
  EventQ.logger.info("[#{self.class}] - Message rejected removing from queue.")
69
- #reject the message to remove from queue
68
+ # reject the message to remove from queue
70
69
  channel.reject(delivery_tag, false)
71
70
 
72
- #check if the message retry limit has been exceeded
71
+ # check if the message retry limit has been exceeded
73
72
  if message.retry_attempts >= queue.max_retry_attempts
74
-
75
73
  EventQ.logger.info("[#{self.class}] - Message retry attempt limit exceeded. Msg: #{serialize_message(message)}")
76
74
 
77
75
  context.call_on_retry_exceeded_block(message)
78
-
79
- #check if the message is allowed to be retried
76
+ # check if the message is allowed to be retried
80
77
  elsif queue.allow_retry
81
-
82
- EventQ.logger.debug { "[#{self.class}] - Incrementing retry attempts count." }
83
78
  message.retry_attempts += 1
79
+ retry_attempts = message.retry_attempts - queue.retry_back_off_grace
80
+ retry_attempts = 1 if retry_attempts < 1
84
81
 
85
82
  if queue.allow_retry_back_off == true
86
- EventQ.logger.debug { "[#{self.class}] - Calculating message back off retry delay. Attempts: #{message.retry_attempts} * Retry Delay: #{queue.retry_delay}" }
87
- message_ttl = message.retry_attempts * queue.retry_delay
88
- if (message.retry_attempts * queue.retry_delay) > queue.max_retry_delay
83
+ message_ttl = retry_attempts * queue.retry_delay
84
+ if (retry_attempts * queue.retry_delay) > queue.max_retry_delay
89
85
  EventQ.logger.debug { "[#{self.class}] - Max message back off retry delay reached." }
90
86
  message_ttl = queue.max_retry_delay
91
87
  end
92
88
  else
93
- EventQ.logger.debug { "[#{self.class}] - Setting fixed retry delay for message." }
94
89
  message_ttl = queue.retry_delay
95
90
  end
96
91
 
97
92
  EventQ.logger.debug { "[#{self.class}] - Sending message for retry. Message TTL: #{message_ttl}" }
98
93
  retry_exchange.publish(serialize_message(message), :expiration => message_ttl)
99
- EventQ.logger.debug { "[#{self.class}] - Published message to retry exchange." }
100
94
 
101
95
  context.call_on_retry_block(message)
102
-
103
96
  end
104
97
 
105
98
  return true
106
-
107
99
  end
108
100
 
109
101
  def configure(options = {})
110
102
  options[:durable] ||= true
111
103
  end
112
104
 
105
+ # Logic for the RabbitMq adapter when a message is accepted
106
+ def acknowledge_message(channel, delivery_tag)
107
+ channel.acknowledge(delivery_tag, false)
108
+ end
109
+
113
110
  private
114
111
 
115
112
  def process_message(payload, queue, channel, retry_exchange, delivery_tag, block)
116
- abort = false
117
- error = false
118
113
  message = deserialize_message(payload)
119
-
120
- EventQ.logger.info("[#{self.class}] - Message received. Retry Attempts: #{message.retry_attempts}")
114
+ retry_attempts = message.retry_attempts
121
115
 
122
116
  @signature_provider_manager.validate_signature(message: message, queue: queue)
123
117
 
124
- message_args = EventQ::MessageArgs.new(
125
- type: message.type,
126
- retry_attempts: message.retry_attempts,
127
- context: message.context,
128
- content_type: message.content_type,
129
- id: message.id,
130
- sent: message.created
131
- )
132
-
133
- if(!EventQ::NonceManager.is_allowed?(message.id))
134
- EventQ.logger.info("[#{self.class}] - Duplicate Message received. Dropping message.")
135
- channel.acknowledge(delivery_tag, false)
136
- return false
137
- end
138
-
139
- # begin worker block for queue message
140
- begin
141
- block.call(message.content, message_args)
118
+ status, message_args = context.process_message(block, message, retry_attempts, [channel, delivery_tag])
142
119
 
143
- if message_args.abort == true
144
- abort = true
145
- EventQ.logger.info("[#{self.class}] - Message aborted.")
146
- else
147
- # accept the message as processed
120
+ case status
121
+ when :duplicate
148
122
  channel.acknowledge(delivery_tag, false)
149
- EventQ.logger.info("[#{self.class}] - Message acknowledged.")
150
- end
151
-
152
- rescue => e
153
- EventQ.logger.error("[#{self.class}] - An unhandled error happened attempting to process a queue message. Error: #{e} | Backtrace: #{e.backtrace}")
154
- error = true
155
- context.call_on_error_block(error: e, message: message)
156
- end
157
-
158
- if error || abort
159
- EventQ::NonceManager.failed(message.id)
160
- reject_message(channel, message, delivery_tag, retry_exchange, queue, abort)
161
- else
162
- EventQ::NonceManager.complete(message.id)
123
+ when :accepted
124
+ # Acceptance was handled directly when QueueWorker#process_message was called
125
+ when :reject
126
+ reject_message(channel, message, delivery_tag, retry_exchange, queue, message_args.abort)
127
+ else
128
+ raise "Unrecognized status: #{status}"
163
129
  end
164
130
  end
165
131
  end
@@ -54,7 +54,7 @@ module EventQ
54
54
  # So each fork gets its own copy of the @worker_status variable.
55
55
  track_process(pid)
56
56
  end
57
-
57
+
58
58
  Process.waitall
59
59
  else
60
60
  # No need to track process/threads separately as we are in the main parent process,
@@ -108,6 +108,61 @@ module EventQ
108
108
  raise Exceptions::WorkerThreadError, e.message, e.backtrace
109
109
  end
110
110
 
111
+ # Method to be called by an adapter. This defines the common logic for processing a message.
112
+ # @param [Array] acceptance_args list of arguments that would be used to accept a message by an adapter.
113
+ # @return [Symbol, MessageArgs] :accepted, :duplicate, :reject
114
+ def process_message(block, message, retry_attempts, acceptance_args)
115
+ abort = false
116
+ error = false
117
+ status = nil
118
+
119
+ message_args = EventQ::MessageArgs.new(
120
+ type: message.type,
121
+ retry_attempts: retry_attempts,
122
+ context: message.context,
123
+ content_type: message.content_type,
124
+ id: message.id,
125
+ sent: message.created
126
+ )
127
+
128
+ EventQ.logger.info("[#{self.class}] - Message received. Retry Attempts: #{retry_attempts}")
129
+
130
+ if (!EventQ::NonceManager.is_allowed?(message.id))
131
+ EventQ.logger.info("[#{self.class}] - Duplicate Message received. Ignoring message.")
132
+ status = :duplicate
133
+ return status, message_args
134
+ end
135
+
136
+ # begin worker block for queue message
137
+ begin
138
+ block.call(message.content, message_args)
139
+
140
+ if message_args.abort == true
141
+ abort = true
142
+ EventQ.logger.info("[#{self.class}] - Message aborted.")
143
+ else
144
+ # accept the message as processed
145
+ status = :accepted
146
+ worker_adapter.acknowledge_message(*acceptance_args)
147
+ EventQ.logger.info("[#{self.class}] - Message acknowledged.")
148
+ end
149
+ rescue => e
150
+ EventQ.logger.error("[#{self.class}] - Unhandled error while attempting to process a queue message.")
151
+ EventQ.logger.error(e)
152
+ error = true
153
+ call_on_error_block(error: e, message: message)
154
+ end
155
+
156
+ if error || abort
157
+ EventQ::NonceManager.failed(message.id)
158
+ status = :reject
159
+ else
160
+ EventQ::NonceManager.complete(message.id)
161
+ end
162
+
163
+ [status, message_args]
164
+ end
165
+
111
166
  def stop
112
167
  EventQ.logger.info("[#{self.class}] - Stopping.")
113
168
  @is_running = false
@@ -1,20 +1,12 @@
1
1
  # require 'eventq_base'
2
2
 
3
- if RUBY_PLATFORM =~ /java/
4
- require 'march_hare'
5
- else
6
- require 'bunny'
7
- end
3
+ require 'bunny'
8
4
 
9
5
  require 'hash_kit'
10
6
  require_relative './eventq_rabbitmq/rabbitmq_queue_client'
11
7
  require_relative './eventq_rabbitmq/rabbitmq_queue_manager'
12
8
 
13
- if RUBY_PLATFORM =~ /java/
14
- require_relative './eventq_rabbitmq/jruby/rabbitmq_queue_worker'
15
- else
16
- require_relative './eventq_rabbitmq/rabbitmq_queue_worker'
17
- end
9
+ require_relative './eventq_rabbitmq/rabbitmq_queue_worker'
18
10
 
19
11
  require_relative './eventq_rabbitmq/rabbitmq_subscription_manager'
20
12
  require_relative './eventq_rabbitmq/rabbitmq_eventq_client'