ruby_kafka_retry 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 1b4796994843dcde9707e29b893adc82b39a628f05dac40b912d21155aa83975
4
+ data.tar.gz: a4d38bc48c0dc253e821e418b3ee0be90ca7bb91c38251bedeb2403f765ecf30
5
+ SHA512:
6
+ metadata.gz: 457a6c19474dac353281f95f44df14601e455efa847a619a066c39ae040939481cc3d98d91ca61f1b5e39e9b77eab416659cdbcf1005c4878a3ada2ca971b786
7
+ data.tar.gz: 8f01eef29a902cfb21a35075efd0646a86b3f34d91e8a39b4790878cd0252b0deea9f30ffb9cec9bc271f8b5e8354dd5a65cf1603711c42a33be7d1cb164a5cc
data/README.md ADDED
@@ -0,0 +1,120 @@
1
+ # RubyKafkaRetry
2
+
3
+ The `ruby_kafka_retry` gem provides a mechanism to handle message retries and dead-letter queue (DLQ) functionality in Ruby applications using Kafka. It ensures messages are retried with an increasing delay before being sent to a DLQ.
4
+
5
+ ## Installation
6
+ Add this token to bundle config for the gem installation access
7
+
8
+ ```sh
9
+ bundle config rubygems.pkg.github.com veeraveeraveera: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
10
+ ```
11
+
12
+ Add this line to your application's Gemfile:
13
+
14
+ ```ruby
15
+ source "https://rubygems.pkg.github.com/veeraveeraveera" do
16
+ gem "ruby_kafka_retry", "0.1.0"
17
+ end
18
+ ```
19
+
20
+ And then execute:
21
+
22
+ $ bundle install
23
+
24
+ <!-- Or install it yourself as:
25
+
26
+ $ gem install ruby_kafka_retry -->
27
+
28
+ ## Usage
29
+ # Retrying Messages
30
+ To handle message retries, use the `RubyKafkaRetry::RetryFailedEvent` class. This class allows you to specify the retry topic, DLQ topic, and the message to be retried, along with an optional maximum retry attempt count.
31
+
32
+ ```ruby
33
+ retry_topic = 'my_retry_topic'
34
+ dlq_topic = 'my_dlq_topic'
35
+ topic_message = { key: 'value' } # The message to be processed
36
+ max_retry_attempt = 5 # Optional parameter, default is 3 if not provided
37
+
38
+ retry_event = RubyKafkaRetry::RetryFailedEvent.new(retry_topic, dlq_topic, topic_message, max_retry_attempt)
39
+ retry_event.retry
40
+ ```
41
+
42
+ # Detailed Description
43
+ 1. **First Retry Attempt**: If the topic_message does not include the `current_retry_attempt` key, the gem considers it as the first retry attempt and `current_retry_attempt` will be appended to the `topic_message` with the value as 1. The modified `topic_message` will then be published to the retry_topic.
44
+ 2. **Message Format**: The `topic_message` must be a hash. If a non-hash object is passed, the gem will raise an error:
45
+ ```ruby
46
+ raise TypeError, 'topic_message must be a Hash'
47
+ ```
48
+ 3. **Retry Logic**:
49
+ * If the `current_retry_attempt` value in the topic_message reaches the `max_retry_attempt` count, the message will be published to the DLQ topic.
50
+ * If the `current_retry_attempt` value is less than the `max_retry_attempt`, the `current_retry_attempt` value will be incremented, and the message will be republished to the `retry_topic` after a delay.
51
+ * The delay before republishing is calculated as `2 ** current_retry_attempt` minutes.
52
+ * The `max_retry_attempt` parameter is optional. If it is not provided, the default value is `3`.
53
+
54
+ # Example Workflow
55
+ Here's a step-by-step example workflow:
56
+ 1. A message `topic_message = { key: 'value' }` is received and processed.
57
+ 2. If processing fails, it triggers a retry:
58
+ * `current_retry_attempt` key is added to the message if not present.
59
+ * Message becomes `{ key: 'value', current_retry_attempt: 1 }`.
60
+ 3. The message is published to the `retry_topic` after a delay of 2 ** 1 (2 minutes).
61
+ 4. If processing fails again, current_retry_attempt is incremented, and the message is republished after a delay of 2 ** 2 (4 minutes).
62
+ 5. This continues until `current_retry_attempt` reaches `max_retry_attempt`.
63
+ 6. Once `max_retry_attempt` is reached, the message is published to the DLQ topic.
64
+
65
+ # Configuration
66
+ You need to configure the gem by creating a YAML configuration file at `config/ruby_kafka_retry.yml`. This file should contain the following settings:
67
+
68
+ ```ruby
69
+ development:
70
+ client_id: "my_kafka_client_id"
71
+ brokers:
72
+ - "localhost:9092"
73
+ ssl_ca_certs_from_system: false
74
+ redis_host: "127.0.0.1"
75
+ redis_db: "10"
76
+ redis_port: "6379"
77
+ sidekiq_queue: "test_retry_queue"
78
+
79
+ stage:
80
+ client_id: "my_kafka_client_id"
81
+ brokers:
82
+ - "localhost:9092"
83
+ ssl_ca_certs_from_system: false
84
+ redis_host: "127.0.0.1"
85
+ redis_db: "10"
86
+ redis_port: "6379"
87
+ sidekiq_queue: "test_retry_queue"
88
+
89
+ production:
90
+ client_id: "my_kafka_client_id"
91
+ brokers:
92
+ - "localhost:9092"
93
+ ssl_ca_certs_from_system: false
94
+ redis_host: "127.0.0.1"
95
+ redis_db: "10"
96
+ redis_port: "6379"
97
+ sidekiq_queue: "test_retry_queue"
98
+ ```
99
+
100
+ add the same `sidekiq_queue` in `sidekiq.yml` file as well
101
+
102
+ # Dependencies
103
+ The `ruby_kafka_retry` gem depends on the following gems:
104
+ * ruby-kafka
105
+ * sidekiq
106
+
107
+ # Running Services
108
+ To use this gem, ensure the following services are running in the background:
109
+ 1. **Kafka Server**: Ensure your Kafka server is up and running.
110
+ 2. **Sidekiq Server**: Start your Sidekiq server to handle background job processing.
111
+
112
+ ## Development
113
+
114
+ After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
115
+
116
+ To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
117
+
118
+ ## Contributing
119
+
120
+ Bug reports and pull requests are welcome on GitHub at https://github.com/veeraveeraveera/ruby_kafka_retry.
@@ -0,0 +1,10 @@
1
+ require_relative "kafka_producer"
2
+
3
+ class FailedEventDelayedRetryWorker
4
+ include Sidekiq::Worker
5
+
6
+ def perform(topic, message)
7
+ KafkaProducer.new.publish_to_topic(topic, message)
8
+ end
9
+
10
+ end
@@ -0,0 +1,15 @@
1
+ class KafkaProducer
2
+
3
+ def initialize
4
+ @configs = YAML.load_file("#{Rails.root}/config/ruby_kafka_retry.yml")[Rails.env]
5
+ end
6
+
7
+ def get_kafka_instance
8
+ Kafka.new(@configs["brokers"], client_id: @configs["client_id"], ssl_ca_certs_from_system: @configs["ssl_ca_certs_from_system"])
9
+ end
10
+
11
+ def publish_to_topic(topic, message)
12
+ kafka = get_kafka_instance
13
+ kafka.deliver_message(message.to_json, topic: topic)
14
+ end
15
+ end
@@ -0,0 +1,5 @@
1
+ # frozen_string_literal: true
2
+
3
+ module RubyKafkaRetry
4
+ VERSION = "0.1.0"
5
+ end
@@ -0,0 +1,49 @@
1
+ # frozen_string_literal: true
2
+
3
+ require_relative "ruby_kafka_retry/version"
4
+ require_relative "ruby_kafka_retry_validator"
5
+ require_relative "kafka_producer"
6
+ require_relative "sidekiq_publisher"
7
+
8
+ module RubyKafkaRetry
9
+ class Error < StandardError; end
10
+ class RetryFailedEvent
11
+
12
+ def initialize(retry_topic, dlq_topic, topic_message, max_retry_attempt=3)
13
+ @retry_topic = retry_topic
14
+ @dlq_topic = dlq_topic
15
+ @topic_message = topic_message
16
+ @max_retry_attempt = max_retry_attempt
17
+ end
18
+
19
+ def validate_params
20
+ RubyKafkaRetryValidator.new(@retry_topic, @dlq_topic, @topic_message).validate
21
+ end
22
+
23
+ def can_add_dlq_topic?(curr_retry_attempt)
24
+ @max_retry_attempt < curr_retry_attempt
25
+ end
26
+
27
+ def format_message
28
+ @topic_message['current_retry_attempt'] = @topic_message['current_retry_attempt'].to_i + 1
29
+ @topic_message
30
+ end
31
+
32
+ def process_message
33
+ message = format_message
34
+ if(can_add_dlq_topic?(message['current_retry_attempt']))
35
+ KafkaProducer.new.publish_to_topic(@dlq_topic, message)
36
+ else
37
+ SidekiqPublisher.new.publish_to_sidekiq(@retry_topic, message)
38
+ end
39
+ end
40
+
41
+ def retry
42
+ validate_params
43
+ process_message
44
+ return true
45
+ end
46
+
47
+ end
48
+
49
+ end
@@ -0,0 +1,34 @@
1
+ class RubyKafkaRetryValidator
2
+ include ActiveModel::Validations
3
+
4
+ attr_accessor :original_topic, :retry_topic, :dlq_topic, :message
5
+
6
+ validate :validate_retry_topic
7
+ validate :validate_dlq_topic
8
+ validate :validate_message
9
+
10
+ def initialize(retry_topic, dlq_topic, message)
11
+ @retry_topic = retry_topic
12
+ @dlq_topic = dlq_topic
13
+ @message = message
14
+ end
15
+
16
+ def validate
17
+ validate_retry_topic
18
+ validate_dlq_topic
19
+ validate_message
20
+ end
21
+
22
+ def validate_retry_topic
23
+ raise "Validation failed: Retry topic shouldn't be blank" unless retry_topic.present?
24
+ end
25
+
26
+ def validate_dlq_topic
27
+ raise "Validation failed: DLQ topic shouldn't be blank" unless dlq_topic.present?
28
+ end
29
+
30
+ def validate_message
31
+ raise "Validation failed: Topic message must be a non-empty Hash" unless(message.present? && message.is_a?(Hash))
32
+ end
33
+
34
+ end
@@ -0,0 +1,27 @@
1
+ require_relative "failed_event_delayed_retry_worker"
2
+
3
+ class SidekiqPublisher
4
+
5
+ def initialize
6
+ @configs = YAML.load_file("#{Rails.root}/config/ruby_kafka_retry.yml")[Rails.env]
7
+ end
8
+
9
+ def configure_sidekiq_server
10
+ Sidekiq.configure_server do |config|
11
+ config.redis = { url: 'redis://' + @configs['redis_host'] + ':' + @configs['redis_port'].to_s + '/' + @configs['redis_db'].to_s}
12
+ end
13
+ Sidekiq.configure_client do |config|
14
+ config.redis = { url: 'redis://' + @configs['redis_host'] + ':' + @configs['redis_port'].to_s + '/' + @configs['redis_db'].to_s}
15
+ end
16
+ end
17
+
18
+ def get_delay_time(current_retry_attempt)
19
+ 2 ** current_retry_attempt
20
+ end
21
+
22
+ def publish_to_sidekiq(topic, message)
23
+ configure_sidekiq_server
24
+ delay_time = get_delay_time(message['current_retry_attempt'])
25
+ FailedEventDelayedRetryWorker.set(queue: @configs['sidekiq_queue']).perform_in(delay_time.minutes, topic, message)
26
+ end
27
+ end
@@ -0,0 +1,35 @@
1
+ # frozen_string_literal: true
2
+
3
+ require_relative "lib/ruby_kafka_retry/version"
4
+
5
+ Gem::Specification.new do |spec|
6
+ spec.name = "ruby_kafka_retry"
7
+ spec.version = RubyKafkaRetry::VERSION
8
+ spec.authors = ["veeramani"]
9
+ spec.email = ["veeramani451998@gmail.com"]
10
+
11
+ spec.summary = "Kafka messages are retried with an increasing delay before being sent to a DLQ"
12
+ spec.description = "The RubyKafkaRetry gem provides a mechanism to handle message retries and dead-letter queue (DLQ) functionality in Ruby applications using Kafka. It ensures messages are retried with an increasing delay before being sent to a DLQ."
13
+ spec.homepage = "https://github.com/veeraveeraveera/ruby_kafka_retry"
14
+ spec.required_ruby_version = ">= 2.2"
15
+
16
+ spec.metadata["homepage_uri"] = "https://github.com/veeraveeraveera/ruby_kafka_retry"
17
+ spec.metadata["source_code_uri"] = "https://github.com/veeraveeraveera/ruby_kafka_retry"
18
+ spec.metadata["changelog_uri"] = "https://github.com/veeraveeraveera/ruby_kafka_retry"
19
+ spec.metadata["rubygems_mfa_required"] = 'true'
20
+
21
+ spec.files = Dir["lib/**/*.rb"] + Dir["exe/*"] + Dir["*.gemspec"] + Dir["README*"]
22
+ spec.bindir = "exe"
23
+ spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
24
+ spec.require_paths = ["lib"]
25
+ spec.add_dependency "rails", ">= 4.2"
26
+ spec.add_dependency "sidekiq"
27
+ spec.add_dependency "ruby-kafka"
28
+ spec.license = "MIT"
29
+
30
+ # Uncomment to register a new dependency of your gem
31
+ # spec.add_dependency "example-gem", "~> 1.0"
32
+
33
+ # For more information and examples about making a new gem, check out our
34
+ # guide at: https://bundler.io/guides/creating_gem.html
35
+ end
metadata ADDED
@@ -0,0 +1,100 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: ruby_kafka_retry
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.1.0
5
+ platform: ruby
6
+ authors:
7
+ - veeramani
8
+ autorequire:
9
+ bindir: exe
10
+ cert_chain: []
11
+ date: 2024-06-29 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: rails
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - ">="
18
+ - !ruby/object:Gem::Version
19
+ version: '4.2'
20
+ type: :runtime
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - ">="
25
+ - !ruby/object:Gem::Version
26
+ version: '4.2'
27
+ - !ruby/object:Gem::Dependency
28
+ name: sidekiq
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - ">="
32
+ - !ruby/object:Gem::Version
33
+ version: '0'
34
+ type: :runtime
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - ">="
39
+ - !ruby/object:Gem::Version
40
+ version: '0'
41
+ - !ruby/object:Gem::Dependency
42
+ name: ruby-kafka
43
+ requirement: !ruby/object:Gem::Requirement
44
+ requirements:
45
+ - - ">="
46
+ - !ruby/object:Gem::Version
47
+ version: '0'
48
+ type: :runtime
49
+ prerelease: false
50
+ version_requirements: !ruby/object:Gem::Requirement
51
+ requirements:
52
+ - - ">="
53
+ - !ruby/object:Gem::Version
54
+ version: '0'
55
+ description: The RubyKafkaRetry gem provides a mechanism to handle message retries
56
+ and dead-letter queue (DLQ) functionality in Ruby applications using Kafka. It ensures
57
+ messages are retried with an increasing delay before being sent to a DLQ.
58
+ email:
59
+ - veeramani451998@gmail.com
60
+ executables: []
61
+ extensions: []
62
+ extra_rdoc_files: []
63
+ files:
64
+ - README.md
65
+ - lib/failed_event_delayed_retry_worker.rb
66
+ - lib/kafka_producer.rb
67
+ - lib/ruby_kafka_retry.rb
68
+ - lib/ruby_kafka_retry/version.rb
69
+ - lib/ruby_kafka_retry_validator.rb
70
+ - lib/sidekiq_publisher.rb
71
+ - ruby_kafka_retry.gemspec
72
+ homepage: https://github.com/veeraveeraveera/ruby_kafka_retry
73
+ licenses:
74
+ - MIT
75
+ metadata:
76
+ homepage_uri: https://github.com/veeraveeraveera/ruby_kafka_retry
77
+ source_code_uri: https://github.com/veeraveeraveera/ruby_kafka_retry
78
+ changelog_uri: https://github.com/veeraveeraveera/ruby_kafka_retry
79
+ rubygems_mfa_required: 'true'
80
+ post_install_message:
81
+ rdoc_options: []
82
+ require_paths:
83
+ - lib
84
+ required_ruby_version: !ruby/object:Gem::Requirement
85
+ requirements:
86
+ - - ">="
87
+ - !ruby/object:Gem::Version
88
+ version: '2.2'
89
+ required_rubygems_version: !ruby/object:Gem::Requirement
90
+ requirements:
91
+ - - ">="
92
+ - !ruby/object:Gem::Version
93
+ version: '0'
94
+ requirements: []
95
+ rubygems_version: 3.3.7
96
+ signing_key:
97
+ specification_version: 4
98
+ summary: Kafka messages are retried with an increasing delay before being sent to
99
+ a DLQ
100
+ test_files: []