ruby_kafka_retry 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/README.md +120 -0
- data/lib/failed_event_delayed_retry_worker.rb +10 -0
- data/lib/kafka_producer.rb +15 -0
- data/lib/ruby_kafka_retry/version.rb +5 -0
- data/lib/ruby_kafka_retry.rb +49 -0
- data/lib/ruby_kafka_retry_validator.rb +34 -0
- data/lib/sidekiq_publisher.rb +27 -0
- data/ruby_kafka_retry.gemspec +35 -0
- metadata +100 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 1b4796994843dcde9707e29b893adc82b39a628f05dac40b912d21155aa83975
|
4
|
+
data.tar.gz: a4d38bc48c0dc253e821e418b3ee0be90ca7bb91c38251bedeb2403f765ecf30
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 457a6c19474dac353281f95f44df14601e455efa847a619a066c39ae040939481cc3d98d91ca61f1b5e39e9b77eab416659cdbcf1005c4878a3ada2ca971b786
|
7
|
+
data.tar.gz: 8f01eef29a902cfb21a35075efd0646a86b3f34d91e8a39b4790878cd0252b0deea9f30ffb9cec9bc271f8b5e8354dd5a65cf1603711c42a33be7d1cb164a5cc
|
data/README.md
ADDED
@@ -0,0 +1,120 @@
|
|
1
|
+
# RubyKafkaRetry
|
2
|
+
|
3
|
+
The `ruby_kafka_retry` gem provides a mechanism to handle message retries and dead-letter queue (DLQ) functionality in Ruby applications using Kafka. It ensures messages are retried with an increasing delay before being sent to a DLQ.
|
4
|
+
|
5
|
+
## Installation
|
6
|
+
Add this token to bundle config for the gem installation access
|
7
|
+
|
8
|
+
```sh
|
9
|
+
bundle config rubygems.pkg.github.com veeraveeraveera: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
|
10
|
+
```
|
11
|
+
|
12
|
+
Add this line to your application's Gemfile:
|
13
|
+
|
14
|
+
```ruby
|
15
|
+
source "https://rubygems.pkg.github.com/veeraveeraveera" do
|
16
|
+
gem "ruby_kafka_retry", "0.1.0"
|
17
|
+
end
|
18
|
+
```
|
19
|
+
|
20
|
+
And then execute:
|
21
|
+
|
22
|
+
$ bundle install
|
23
|
+
|
24
|
+
<!-- Or install it yourself as:
|
25
|
+
|
26
|
+
$ gem install ruby_kafka_retry -->
|
27
|
+
|
28
|
+
## Usage
|
29
|
+
# Retrying Messages
|
30
|
+
To handle message retries, use the `RubyKafkaRetry::RetryFailedEvent` class. This class allows you to specify the retry topic, DLQ topic, and the message to be retried, along with an optional maximum retry attempt count.
|
31
|
+
|
32
|
+
```ruby
|
33
|
+
retry_topic = 'my_retry_topic'
|
34
|
+
dlq_topic = 'my_dlq_topic'
|
35
|
+
topic_message = { key: 'value' } # The message to be processed
|
36
|
+
max_retry_attempt = 5 # Optional parameter, default is 3 if not provided
|
37
|
+
|
38
|
+
retry_event = RubyKafkaRetry::RetryFailedEvent.new(retry_topic, dlq_topic, topic_message, max_retry_attempt)
|
39
|
+
retry_event.retry
|
40
|
+
```
|
41
|
+
|
42
|
+
# Detailed Description
|
43
|
+
1. **First Retry Attempt**: If the topic_message does not include the `current_retry_attempt` key, the gem considers it as the first retry attempt and `current_retry_attempt` will be appended to the `topic_message` with the value as 1. The modified `topic_message` will then be published to the retry_topic.
|
44
|
+
2. **Message Format**: The `topic_message` must be a hash. If a non-hash object is passed, the gem will raise an error:
|
45
|
+
```ruby
|
46
|
+
raise TypeError, 'topic_message must be a Hash'
|
47
|
+
```
|
48
|
+
3. **Retry Logic**:
|
49
|
+
* If the `current_retry_attempt` value in the topic_message reaches the `max_retry_attempt` count, the message will be published to the DLQ topic.
|
50
|
+
* If the `current_retry_attempt` value is less than the `max_retry_attempt`, the `current_retry_attempt` value will be incremented, and the message will be republished to the `retry_topic` after a delay.
|
51
|
+
* The delay before republishing is calculated as `2 ** current_retry_attempt` minutes.
|
52
|
+
* The `max_retry_attempt` parameter is optional. If it is not provided, the default value is `3`.
|
53
|
+
|
54
|
+
# Example Workflow
|
55
|
+
Here's a step-by-step example workflow:
|
56
|
+
1. A message `topic_message = { key: 'value' }` is received and processed.
|
57
|
+
2. If processing fails, it triggers a retry:
|
58
|
+
* `current_retry_attempt` key is added to the message if not present.
|
59
|
+
* Message becomes `{ key: 'value', current_retry_attempt: 1 }`.
|
60
|
+
3. The message is published to the `retry_topic` after a delay of 2 ** 1 (2 minutes).
|
61
|
+
4. If processing fails again, current_retry_attempt is incremented, and the message is republished after a delay of 2 ** 2 (4 minutes).
|
62
|
+
5. This continues until `current_retry_attempt` reaches `max_retry_attempt`.
|
63
|
+
6. Once `max_retry_attempt` is reached, the message is published to the DLQ topic.
|
64
|
+
|
65
|
+
# Configuration
|
66
|
+
You need to configure the gem by creating a YAML configuration file at `config/ruby_kafka_retry.yml`. This file should contain the following settings:
|
67
|
+
|
68
|
+
```ruby
|
69
|
+
development:
|
70
|
+
client_id: "my_kafka_client_id"
|
71
|
+
brokers:
|
72
|
+
- "localhost:9092"
|
73
|
+
ssl_ca_certs_from_system: false
|
74
|
+
redis_host: "127.0.0.1"
|
75
|
+
redis_db: "10"
|
76
|
+
redis_port: "6379"
|
77
|
+
sidekiq_queue: "test_retry_queue"
|
78
|
+
|
79
|
+
stage:
|
80
|
+
client_id: "my_kafka_client_id"
|
81
|
+
brokers:
|
82
|
+
- "localhost:9092"
|
83
|
+
ssl_ca_certs_from_system: false
|
84
|
+
redis_host: "127.0.0.1"
|
85
|
+
redis_db: "10"
|
86
|
+
redis_port: "6379"
|
87
|
+
sidekiq_queue: "test_retry_queue"
|
88
|
+
|
89
|
+
production:
|
90
|
+
client_id: "my_kafka_client_id"
|
91
|
+
brokers:
|
92
|
+
- "localhost:9092"
|
93
|
+
ssl_ca_certs_from_system: false
|
94
|
+
redis_host: "127.0.0.1"
|
95
|
+
redis_db: "10"
|
96
|
+
redis_port: "6379"
|
97
|
+
sidekiq_queue: "test_retry_queue"
|
98
|
+
```
|
99
|
+
|
100
|
+
add the same `sidekiq_queue` in `sidekiq.yml` file as well
|
101
|
+
|
102
|
+
# Dependencies
|
103
|
+
The `ruby_kafka_retry` gem depends on the following gems:
|
104
|
+
* ruby-kafka
|
105
|
+
* sidekiq
|
106
|
+
|
107
|
+
# Running Services
|
108
|
+
To use this gem, ensure the following services are running in the background:
|
109
|
+
1. **Kafka Server**: Ensure your Kafka server is up and running.
|
110
|
+
2. **Sidekiq Server**: Start your Sidekiq server to handle background job processing.
|
111
|
+
|
112
|
+
## Development
|
113
|
+
|
114
|
+
After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
|
115
|
+
|
116
|
+
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
|
117
|
+
|
118
|
+
## Contributing
|
119
|
+
|
120
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/veeraveeraveera/ruby_kafka_retry.
|
@@ -0,0 +1,15 @@
|
|
1
|
+
class KafkaProducer
|
2
|
+
|
3
|
+
def initialize
|
4
|
+
@configs = YAML.load_file("#{Rails.root}/config/ruby_kafka_retry.yml")[Rails.env]
|
5
|
+
end
|
6
|
+
|
7
|
+
def get_kafka_instance
|
8
|
+
Kafka.new(@configs["brokers"], client_id: @configs["client_id"], ssl_ca_certs_from_system: @configs["ssl_ca_certs_from_system"])
|
9
|
+
end
|
10
|
+
|
11
|
+
def publish_to_topic(topic, message)
|
12
|
+
kafka = get_kafka_instance
|
13
|
+
kafka.deliver_message(message.to_json, topic: topic)
|
14
|
+
end
|
15
|
+
end
|
@@ -0,0 +1,49 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require_relative "ruby_kafka_retry/version"
|
4
|
+
require_relative "ruby_kafka_retry_validator"
|
5
|
+
require_relative "kafka_producer"
|
6
|
+
require_relative "sidekiq_publisher"
|
7
|
+
|
8
|
+
module RubyKafkaRetry
|
9
|
+
class Error < StandardError; end
|
10
|
+
class RetryFailedEvent
|
11
|
+
|
12
|
+
def initialize(retry_topic, dlq_topic, topic_message, max_retry_attempt=3)
|
13
|
+
@retry_topic = retry_topic
|
14
|
+
@dlq_topic = dlq_topic
|
15
|
+
@topic_message = topic_message
|
16
|
+
@max_retry_attempt = max_retry_attempt
|
17
|
+
end
|
18
|
+
|
19
|
+
def validate_params
|
20
|
+
RubyKafkaRetryValidator.new(@retry_topic, @dlq_topic, @topic_message).validate
|
21
|
+
end
|
22
|
+
|
23
|
+
def can_add_dlq_topic?(curr_retry_attempt)
|
24
|
+
@max_retry_attempt < curr_retry_attempt
|
25
|
+
end
|
26
|
+
|
27
|
+
def format_message
|
28
|
+
@topic_message['current_retry_attempt'] = @topic_message['current_retry_attempt'].to_i + 1
|
29
|
+
@topic_message
|
30
|
+
end
|
31
|
+
|
32
|
+
def process_message
|
33
|
+
message = format_message
|
34
|
+
if(can_add_dlq_topic?(message['current_retry_attempt']))
|
35
|
+
KafkaProducer.new.publish_to_topic(@dlq_topic, message)
|
36
|
+
else
|
37
|
+
SidekiqPublisher.new.publish_to_sidekiq(@retry_topic, message)
|
38
|
+
end
|
39
|
+
end
|
40
|
+
|
41
|
+
def retry
|
42
|
+
validate_params
|
43
|
+
process_message
|
44
|
+
return true
|
45
|
+
end
|
46
|
+
|
47
|
+
end
|
48
|
+
|
49
|
+
end
|
@@ -0,0 +1,34 @@
|
|
1
|
+
class RubyKafkaRetryValidator
|
2
|
+
include ActiveModel::Validations
|
3
|
+
|
4
|
+
attr_accessor :original_topic, :retry_topic, :dlq_topic, :message
|
5
|
+
|
6
|
+
validate :validate_retry_topic
|
7
|
+
validate :validate_dlq_topic
|
8
|
+
validate :validate_message
|
9
|
+
|
10
|
+
def initialize(retry_topic, dlq_topic, message)
|
11
|
+
@retry_topic = retry_topic
|
12
|
+
@dlq_topic = dlq_topic
|
13
|
+
@message = message
|
14
|
+
end
|
15
|
+
|
16
|
+
def validate
|
17
|
+
validate_retry_topic
|
18
|
+
validate_dlq_topic
|
19
|
+
validate_message
|
20
|
+
end
|
21
|
+
|
22
|
+
def validate_retry_topic
|
23
|
+
raise "Validation failed: Retry topic shouldn't be blank" unless retry_topic.present?
|
24
|
+
end
|
25
|
+
|
26
|
+
def validate_dlq_topic
|
27
|
+
raise "Validation failed: DLQ topic shouldn't be blank" unless dlq_topic.present?
|
28
|
+
end
|
29
|
+
|
30
|
+
def validate_message
|
31
|
+
raise "Validation failed: Topic message must be a non-empty Hash" unless(message.present? && message.is_a?(Hash))
|
32
|
+
end
|
33
|
+
|
34
|
+
end
|
@@ -0,0 +1,27 @@
|
|
1
|
+
require_relative "failed_event_delayed_retry_worker"
|
2
|
+
|
3
|
+
class SidekiqPublisher
|
4
|
+
|
5
|
+
def initialize
|
6
|
+
@configs = YAML.load_file("#{Rails.root}/config/ruby_kafka_retry.yml")[Rails.env]
|
7
|
+
end
|
8
|
+
|
9
|
+
def configure_sidekiq_server
|
10
|
+
Sidekiq.configure_server do |config|
|
11
|
+
config.redis = { url: 'redis://' + @configs['redis_host'] + ':' + @configs['redis_port'].to_s + '/' + @configs['redis_db'].to_s}
|
12
|
+
end
|
13
|
+
Sidekiq.configure_client do |config|
|
14
|
+
config.redis = { url: 'redis://' + @configs['redis_host'] + ':' + @configs['redis_port'].to_s + '/' + @configs['redis_db'].to_s}
|
15
|
+
end
|
16
|
+
end
|
17
|
+
|
18
|
+
def get_delay_time(current_retry_attempt)
|
19
|
+
2 ** current_retry_attempt
|
20
|
+
end
|
21
|
+
|
22
|
+
def publish_to_sidekiq(topic, message)
|
23
|
+
configure_sidekiq_server
|
24
|
+
delay_time = get_delay_time(message['current_retry_attempt'])
|
25
|
+
FailedEventDelayedRetryWorker.set(queue: @configs['sidekiq_queue']).perform_in(delay_time.minutes, topic, message)
|
26
|
+
end
|
27
|
+
end
|
@@ -0,0 +1,35 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require_relative "lib/ruby_kafka_retry/version"
|
4
|
+
|
5
|
+
Gem::Specification.new do |spec|
|
6
|
+
spec.name = "ruby_kafka_retry"
|
7
|
+
spec.version = RubyKafkaRetry::VERSION
|
8
|
+
spec.authors = ["veeramani"]
|
9
|
+
spec.email = ["veeramani451998@gmail.com"]
|
10
|
+
|
11
|
+
spec.summary = "Kafka messages are retried with an increasing delay before being sent to a DLQ"
|
12
|
+
spec.description = "The RubyKafkaRetry gem provides a mechanism to handle message retries and dead-letter queue (DLQ) functionality in Ruby applications using Kafka. It ensures messages are retried with an increasing delay before being sent to a DLQ."
|
13
|
+
spec.homepage = "https://github.com/veeraveeraveera/ruby_kafka_retry"
|
14
|
+
spec.required_ruby_version = ">= 2.2"
|
15
|
+
|
16
|
+
spec.metadata["homepage_uri"] = "https://github.com/veeraveeraveera/ruby_kafka_retry"
|
17
|
+
spec.metadata["source_code_uri"] = "https://github.com/veeraveeraveera/ruby_kafka_retry"
|
18
|
+
spec.metadata["changelog_uri"] = "https://github.com/veeraveeraveera/ruby_kafka_retry"
|
19
|
+
spec.metadata["rubygems_mfa_required"] = 'true'
|
20
|
+
|
21
|
+
spec.files = Dir["lib/**/*.rb"] + Dir["exe/*"] + Dir["*.gemspec"] + Dir["README*"]
|
22
|
+
spec.bindir = "exe"
|
23
|
+
spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
|
24
|
+
spec.require_paths = ["lib"]
|
25
|
+
spec.add_dependency "rails", ">= 4.2"
|
26
|
+
spec.add_dependency "sidekiq"
|
27
|
+
spec.add_dependency "ruby-kafka"
|
28
|
+
spec.license = "MIT"
|
29
|
+
|
30
|
+
# Uncomment to register a new dependency of your gem
|
31
|
+
# spec.add_dependency "example-gem", "~> 1.0"
|
32
|
+
|
33
|
+
# For more information and examples about making a new gem, check out our
|
34
|
+
# guide at: https://bundler.io/guides/creating_gem.html
|
35
|
+
end
|
metadata
ADDED
@@ -0,0 +1,100 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: ruby_kafka_retry
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.1.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- veeramani
|
8
|
+
autorequire:
|
9
|
+
bindir: exe
|
10
|
+
cert_chain: []
|
11
|
+
date: 2024-06-29 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: rails
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - ">="
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: '4.2'
|
20
|
+
type: :runtime
|
21
|
+
prerelease: false
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - ">="
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: '4.2'
|
27
|
+
- !ruby/object:Gem::Dependency
|
28
|
+
name: sidekiq
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - ">="
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: '0'
|
34
|
+
type: :runtime
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - ">="
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: '0'
|
41
|
+
- !ruby/object:Gem::Dependency
|
42
|
+
name: ruby-kafka
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - ">="
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '0'
|
48
|
+
type: :runtime
|
49
|
+
prerelease: false
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - ">="
|
53
|
+
- !ruby/object:Gem::Version
|
54
|
+
version: '0'
|
55
|
+
description: The RubyKafkaRetry gem provides a mechanism to handle message retries
|
56
|
+
and dead-letter queue (DLQ) functionality in Ruby applications using Kafka. It ensures
|
57
|
+
messages are retried with an increasing delay before being sent to a DLQ.
|
58
|
+
email:
|
59
|
+
- veeramani451998@gmail.com
|
60
|
+
executables: []
|
61
|
+
extensions: []
|
62
|
+
extra_rdoc_files: []
|
63
|
+
files:
|
64
|
+
- README.md
|
65
|
+
- lib/failed_event_delayed_retry_worker.rb
|
66
|
+
- lib/kafka_producer.rb
|
67
|
+
- lib/ruby_kafka_retry.rb
|
68
|
+
- lib/ruby_kafka_retry/version.rb
|
69
|
+
- lib/ruby_kafka_retry_validator.rb
|
70
|
+
- lib/sidekiq_publisher.rb
|
71
|
+
- ruby_kafka_retry.gemspec
|
72
|
+
homepage: https://github.com/veeraveeraveera/ruby_kafka_retry
|
73
|
+
licenses:
|
74
|
+
- MIT
|
75
|
+
metadata:
|
76
|
+
homepage_uri: https://github.com/veeraveeraveera/ruby_kafka_retry
|
77
|
+
source_code_uri: https://github.com/veeraveeraveera/ruby_kafka_retry
|
78
|
+
changelog_uri: https://github.com/veeraveeraveera/ruby_kafka_retry
|
79
|
+
rubygems_mfa_required: 'true'
|
80
|
+
post_install_message:
|
81
|
+
rdoc_options: []
|
82
|
+
require_paths:
|
83
|
+
- lib
|
84
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
85
|
+
requirements:
|
86
|
+
- - ">="
|
87
|
+
- !ruby/object:Gem::Version
|
88
|
+
version: '2.2'
|
89
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
90
|
+
requirements:
|
91
|
+
- - ">="
|
92
|
+
- !ruby/object:Gem::Version
|
93
|
+
version: '0'
|
94
|
+
requirements: []
|
95
|
+
rubygems_version: 3.3.7
|
96
|
+
signing_key:
|
97
|
+
specification_version: 4
|
98
|
+
summary: Kafka messages are retried with an increasing delay before being sent to
|
99
|
+
a DLQ
|
100
|
+
test_files: []
|