ruby_kafka_retry 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 1b4796994843dcde9707e29b893adc82b39a628f05dac40b912d21155aa83975
+  data.tar.gz: a4d38bc48c0dc253e821e418b3ee0be90ca7bb91c38251bedeb2403f765ecf30
+SHA512:
+  metadata.gz: 457a6c19474dac353281f95f44df14601e455efa847a619a066c39ae040939481cc3d98d91ca61f1b5e39e9b77eab416659cdbcf1005c4878a3ada2ca971b786
+  data.tar.gz: 8f01eef29a902cfb21a35075efd0646a86b3f34d91e8a39b4790878cd0252b0deea9f30ffb9cec9bc271f8b5e8354dd5a65cf1603711c42a33be7d1cb164a5cc

data/README.md

@@ -0,0 +1,120 @@
+# RubyKafkaRetry
+
+The `ruby_kafka_retry` gem provides a mechanism to handle message retries and dead-letter queue (DLQ) functionality in Ruby applications using Kafka. It ensures messages are retried with an increasing delay before being sent to a DLQ.
+
+## Installation
+Add this token to bundle config for the gem installation access
+
+```sh
+bundle config rubygems.pkg.github.com veeraveeraveera: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
+```
+
+Add this line to your application's Gemfile:
+
+```ruby
+source "https://rubygems.pkg.github.com/veeraveeraveera" do
+  gem "ruby_kafka_retry", "0.1.0"
+end
+```
+
+And then execute:
+
+    $ bundle install
+
+<!-- Or install it yourself as:
+
+    $ gem install ruby_kafka_retry -->
+
+## Usage
+# Retrying Messages
+To handle message retries, use the `RubyKafkaRetry::RetryFailedEvent` class. This class allows you to specify the retry topic, DLQ topic, and the message to be retried, along with an optional maximum retry attempt count.
+
+```ruby
+retry_topic = 'my_retry_topic'
+dlq_topic = 'my_dlq_topic'
+topic_message = { key: 'value' }  # The message to be processed
+max_retry_attempt = 5  # Optional parameter, default is 3 if not provided
+
+retry_event = RubyKafkaRetry::RetryFailedEvent.new(retry_topic, dlq_topic, topic_message, max_retry_attempt)
+retry_event.retry
+```
+
+# Detailed Description
+1. **First Retry Attempt**: If the topic_message does not include the `current_retry_attempt` key, the gem considers it as the first retry attempt and `current_retry_attempt` will be appended to the `topic_message` with the value as 1. The modified `topic_message` will then be published to the retry_topic.
+2. **Message Format**: The `topic_message` must be a hash. If a non-hash object is passed, the gem will raise an error:
+     ```ruby
+     raise TypeError, 'topic_message must be a Hash'
+     ```
+3. **Retry Logic**:
+    * If the `current_retry_attempt` value in the topic_message reaches the `max_retry_attempt` count, the message will be published to the DLQ topic.
+    * If the `current_retry_attempt` value is less than the `max_retry_attempt`, the `current_retry_attempt` value will be incremented, and the message will be republished to the `retry_topic` after a delay.
+    * The delay before republishing is calculated as `2 ** current_retry_attempt` minutes.
+    * The `max_retry_attempt` parameter is optional. If it is not provided, the default value is `3`.
+
+# Example Workflow
+Here's a step-by-step example workflow:
+1. A message `topic_message = { key: 'value' }` is received and processed.
+2. If processing fails, it triggers a retry:
+    * `current_retry_attempt` key is added to the message if not present.
+    * Message becomes `{ key: 'value', current_retry_attempt: 1 }`.
+3. The message is published to the `retry_topic` after a delay of 2 ** 1 (2 minutes).
+4. If processing fails again, current_retry_attempt is incremented, and the message is republished after a delay of 2 ** 2 (4 minutes).
+5. This continues until `current_retry_attempt` reaches `max_retry_attempt`.
+6. Once `max_retry_attempt` is reached, the message is published to the DLQ topic.
+
+# Configuration
+You need to configure the gem by creating a YAML configuration file at `config/ruby_kafka_retry.yml`. This file should contain the following settings:
+
+```ruby
+development:
+  client_id: "my_kafka_client_id"
+  brokers:
+    - "localhost:9092"
+  ssl_ca_certs_from_system: false
+  redis_host: "127.0.0.1"
+  redis_db: "10"
+  redis_port: "6379"
+  sidekiq_queue: "test_retry_queue"
+
+stage:
+  client_id: "my_kafka_client_id"
+  brokers:
+    - "localhost:9092"
+  ssl_ca_certs_from_system: false
+  redis_host: "127.0.0.1"
+  redis_db: "10"
+  redis_port: "6379"
+  sidekiq_queue: "test_retry_queue"
+
+production:
+  client_id: "my_kafka_client_id"
+  brokers:
+    - "localhost:9092"
+  ssl_ca_certs_from_system: false
+  redis_host: "127.0.0.1"
+  redis_db: "10"
+  redis_port: "6379"
+  sidekiq_queue: "test_retry_queue"
+```
+
+add the same `sidekiq_queue` in `sidekiq.yml` file as well
+
+# Dependencies
+The `ruby_kafka_retry` gem depends on the following gems:
+* ruby-kafka
+* sidekiq
+
+# Running Services
+To use this gem, ensure the following services are running in the background:
+1. **Kafka Server**: Ensure your Kafka server is up and running.
+2. **Sidekiq Server**: Start your Sidekiq server to handle background job processing.
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/veeraveeraveera/ruby_kafka_retry.

data/lib/failed_event_delayed_retry_worker.rb

@@ -0,0 +1,10 @@
+require_relative "kafka_producer"
+
+class FailedEventDelayedRetryWorker
+  include Sidekiq::Worker
+
+  def perform(topic, message)
+    KafkaProducer.new.publish_to_topic(topic, message)
+  end
+
+end

data/lib/kafka_producer.rb

@@ -0,0 +1,15 @@
+class KafkaProducer
+
+  def initialize
+    @configs = YAML.load_file("#{Rails.root}/config/ruby_kafka_retry.yml")[Rails.env]
+  end
+
+  def get_kafka_instance
+    Kafka.new(@configs["brokers"], client_id: @configs["client_id"], ssl_ca_certs_from_system: @configs["ssl_ca_certs_from_system"])
+  end
+
+  def publish_to_topic(topic, message)
+    kafka = get_kafka_instance
+    kafka.deliver_message(message.to_json, topic: topic)
+  end
+end

data/lib/ruby_kafka_retry/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module RubyKafkaRetry
+  VERSION = "0.1.0"
+end

data/lib/ruby_kafka_retry.rb

@@ -0,0 +1,49 @@
+# frozen_string_literal: true
+
+require_relative "ruby_kafka_retry/version"
+require_relative "ruby_kafka_retry_validator"
+require_relative "kafka_producer"
+require_relative "sidekiq_publisher"
+
+module RubyKafkaRetry
+  class Error < StandardError; end
+  class RetryFailedEvent
+
+    def initialize(retry_topic, dlq_topic, topic_message, max_retry_attempt=3)
+      @retry_topic = retry_topic
+      @dlq_topic = dlq_topic
+      @topic_message = topic_message
+      @max_retry_attempt = max_retry_attempt
+    end
+
+    def validate_params
+      RubyKafkaRetryValidator.new(@retry_topic, @dlq_topic, @topic_message).validate
+    end
+
+    def can_add_dlq_topic?(curr_retry_attempt)
+      @max_retry_attempt < curr_retry_attempt
+    end
+
+    def format_message
+      @topic_message['current_retry_attempt'] = @topic_message['current_retry_attempt'].to_i + 1
+      @topic_message
+    end
+
+    def process_message
+      message = format_message
+      if(can_add_dlq_topic?(message['current_retry_attempt']))
+        KafkaProducer.new.publish_to_topic(@dlq_topic, message)
+      else
+        SidekiqPublisher.new.publish_to_sidekiq(@retry_topic, message)
+      end
+    end
+
+    def retry
+      validate_params
+      process_message
+      return true
+    end
+
+  end
+
+end

data/lib/ruby_kafka_retry_validator.rb

@@ -0,0 +1,34 @@
+class RubyKafkaRetryValidator
+  include ActiveModel::Validations
+
+  attr_accessor :original_topic, :retry_topic, :dlq_topic, :message
+
+  validate :validate_retry_topic
+  validate :validate_dlq_topic
+  validate :validate_message
+
+  def initialize(retry_topic, dlq_topic, message)
+    @retry_topic = retry_topic
+    @dlq_topic = dlq_topic
+    @message = message
+  end
+
+  def validate
+    validate_retry_topic
+    validate_dlq_topic
+    validate_message
+  end
+
+  def validate_retry_topic
+    raise "Validation failed: Retry topic shouldn't be blank" unless retry_topic.present?
+  end
+
+  def validate_dlq_topic
+    raise "Validation failed: DLQ topic shouldn't be blank" unless dlq_topic.present?
+  end
+
+  def validate_message
+    raise "Validation failed: Topic message must be a non-empty Hash" unless(message.present? && message.is_a?(Hash))
+  end
+
+end

data/lib/sidekiq_publisher.rb

@@ -0,0 +1,27 @@
+require_relative "failed_event_delayed_retry_worker"
+
+class SidekiqPublisher
+
+  def initialize
+    @configs = YAML.load_file("#{Rails.root}/config/ruby_kafka_retry.yml")[Rails.env]
+  end
+
+  def configure_sidekiq_server
+    Sidekiq.configure_server do |config|
+      config.redis = { url: 'redis://' + @configs['redis_host'] + ':' + @configs['redis_port'].to_s + '/' + @configs['redis_db'].to_s}
+    end
+    Sidekiq.configure_client do |config|
+      config.redis = { url: 'redis://' + @configs['redis_host'] + ':' + @configs['redis_port'].to_s + '/' + @configs['redis_db'].to_s}
+    end
+  end
+
+  def get_delay_time(current_retry_attempt)
+    2 ** current_retry_attempt
+  end
+
+  def publish_to_sidekiq(topic, message)
+    configure_sidekiq_server
+    delay_time = get_delay_time(message['current_retry_attempt'])
+    FailedEventDelayedRetryWorker.set(queue: @configs['sidekiq_queue']).perform_in(delay_time.minutes, topic, message)
+  end
+end

data/ruby_kafka_retry.gemspec

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+require_relative "lib/ruby_kafka_retry/version"
+
+Gem::Specification.new do |spec|
+  spec.name = "ruby_kafka_retry"
+  spec.version = RubyKafkaRetry::VERSION
+  spec.authors = ["veeramani"]
+  spec.email = ["veeramani451998@gmail.com"]
+
+  spec.summary = "Kafka messages are retried with an increasing delay before being sent to a DLQ"
+  spec.description = "The RubyKafkaRetry gem provides a mechanism to handle message retries and dead-letter queue (DLQ) functionality in Ruby applications using Kafka. It ensures messages are retried with an increasing delay before being sent to a DLQ."
+  spec.homepage = "https://github.com/veeraveeraveera/ruby_kafka_retry"
+  spec.required_ruby_version = ">= 2.2"
+
+  spec.metadata["homepage_uri"] = "https://github.com/veeraveeraveera/ruby_kafka_retry"
+  spec.metadata["source_code_uri"] = "https://github.com/veeraveeraveera/ruby_kafka_retry"
+  spec.metadata["changelog_uri"] = "https://github.com/veeraveeraveera/ruby_kafka_retry"
+  spec.metadata["rubygems_mfa_required"] = 'true'
+
+  spec.files = Dir["lib/**/*.rb"] + Dir["exe/*"] + Dir["*.gemspec"] + Dir["README*"]
+  spec.bindir = "exe"
+  spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+  spec.add_dependency "rails", ">= 4.2"
+  spec.add_dependency "sidekiq"
+  spec.add_dependency "ruby-kafka"
+  spec.license = "MIT"
+
+  # Uncomment to register a new dependency of your gem
+  # spec.add_dependency "example-gem", "~> 1.0"
+
+  # For more information and examples about making a new gem, check out our
+  # guide at: https://bundler.io/guides/creating_gem.html
+end

metadata

@@ -0,0 +1,100 @@
+--- !ruby/object:Gem::Specification
+name: ruby_kafka_retry
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- veeramani
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2024-06-29 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '4.2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '4.2'
+- !ruby/object:Gem::Dependency
+  name: sidekiq
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: ruby-kafka
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: The RubyKafkaRetry gem provides a mechanism to handle message retries
+  and dead-letter queue (DLQ) functionality in Ruby applications using Kafka. It ensures
+  messages are retried with an increasing delay before being sent to a DLQ.
+email:
+- veeramani451998@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- README.md
+- lib/failed_event_delayed_retry_worker.rb
+- lib/kafka_producer.rb
+- lib/ruby_kafka_retry.rb
+- lib/ruby_kafka_retry/version.rb
+- lib/ruby_kafka_retry_validator.rb
+- lib/sidekiq_publisher.rb
+- ruby_kafka_retry.gemspec
+homepage: https://github.com/veeraveeraveera/ruby_kafka_retry
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/veeraveeraveera/ruby_kafka_retry
+  source_code_uri: https://github.com/veeraveeraveera/ruby_kafka_retry
+  changelog_uri: https://github.com/veeraveeraveera/ruby_kafka_retry
+  rubygems_mfa_required: 'true'
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '2.2'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.3.7
+signing_key: 
+specification_version: 4
+summary: Kafka messages are retried with an increasing delay before being sent to
+  a DLQ
+test_files: []