rimless 2.8.0 → 3.0.0

data/doc/upgrade-guide-sources/README.md

@@ -0,0 +1,221 @@
+# Top Level Overview
+
+- [Dependency Changes](#dependency-changes)
+- [Structural Changes](#structural-changes)
+  - [Removed](#removed)
+  - [Moved, API unchanged, constants changed](#moved-api-unchanged-constants-changed)
+  - [Moved, API mostly unchanged, constants changed](#moved-api-mostly-unchanged-constants-changed)
+  - [Moved, API unchanged](#moved-api-unchanged)
+  - [Logic changed](#logic-changed)
+- [Configuration Changes](#configuration-changes)
+  - [Changed](#changed)
+  - [New](#new)
+- [Producer (WaterDrop)](#producer-waterdrop)
+- [Consumer Setup (Karafka)](#consumer-setup-karafka)
+  - [Karafka 2 (New)](#karafka-2-new)
+    - [Phase 1: Boot](#phase-1-boot)
+    - [Phase 2: Message Consumption (batch fetching, single consumption)](#phase-2-message-consumption-batch-fetching-single-consumption)
+  - [Karafka 1 (Old)](#karafka-1-old)
+    - [Phase 1: Boot](#phase-1-boot-1)
+    - [Phase 2: Message Consumption (batch fetching, single consumption)](#phase-2-message-consumption-batch-fetching-single-consumption-1)
+- [Testing](#testing)
+
+## Dependency Changes
+
+* Migrated the `avro_turf` gem from `~> 0.11.0` to `~> 1.20`
+* Migrated the `waterdrop` gem from `~> 1.4` to `~> 2.8`
+* Migrated the `karafka` gem from `~> 1.4` to `~> 2.5`
+* Switched from Sidekiq to ActiveJob (and added the dependency)
+
+## Structural Changes
+
+### Removed
+
+* lib/rimless/karafka/base64_interchanger.rb
+  `Rimless::Karafka::Base64Interchanger` (no interchanging needed, as we do not
+  move the avro/binary encoded kafka message payload from Karafka to Sidekiq
+  anymore — instead we decode the message payload within the Karafka process,
+  and move the decoded data to ActiveJob and use the regular ActiveJob
+  arguments serialization)
+* lib/rimless/karafka/passthrough_mapper.rb
+  `Rimless::Karafka::PassthroughMapper` (Karafka removed consumer/topics
+  mapper concepts completely — which was effectively a no-op, as Rimless
+  implemented a passthrough mapper to just keep inputs equal to outputs for
+  these names)
+* lib/rimless/compatibility/karafka_1_4.rb (dropped Karafka 1.x support)
+
+### Moved, API unchanged, constants changed
+
+* lib/rimless/karafka/avro_deserializer.rb `Rimless::Karafka::AvroDeserializer`
+  -> lib/rimless/consumer/avro_deserializer.rb
+  `Rimless::Consumer::AvroDeserializer`
+
+### Moved, API mostly unchanged, constants changed
+
+* lib/rimless/consumer.rb `Rimless::ConsumerApp` -> lib/rimless/consumer/app.rb
+  `Rimless::Consumer::App` (functionality is kept, but some methods were
+  removed)
+* lib/rimless/consumer_job.rb `Rimless::ConsumerJob` ->
+  lib/rimless/consumer/job.rb `Rimless::Consumer::Job` (Sidekiq -> ActiveJob)
+* lib/rimless/base_consumer.rb `Rimless::BaseConsumer` ->
+  lib/rimless/consumer/base.rb `Rimless::Consumer::Base` (all
+  functionality/methods kept, API extended, `#consume` now returns the
+  `#messages` array instead of the result of the dispatched event method)
+
+### Moved, API unchanged
+
+* lib/rimless/avro_helpers.rb -> lib/rimless/extensions/avro_helpers.rb
+* lib/rimless/configuration_handling.rb ->
+  lib/rimless/extensions/configuration_handling.rb
+* lib/rimless/kafka_helpers.rb -> lib/rimless/extensions/kafka_helpers.rb
+* lib/rimless/dependencies.rb -> lib/rimless/extensions/dependencies.rb
+
+### Logic changed
+
+* lib/rimless/railtie.rb (Karafka is no longer initialized within a Sidekiq
+  server context; this was needed in the past for the encoded/binary Kafka
+  message payload interchanging, as the data was actually parsed within the
+  Sidekiq process)
+
+## Configuration Changes
+
+### Changed
+
+* `KAFKA_BROKERS` (env var) / `config.kafka_brokers` (format change — no
+  protocol anymore, just host:port CSV, old format:
+  `kafka://your.domain:9092,kafka..`, new format: `your.domain:9092,host..`) —
+  the old format is still supported for compatibility
+
+### New
+
+* `config.consumer_logger_listener` (allows configuring the Karafka logging,
+  or providing a custom solution)
+* `config.job_bridge_class` (allows configuring a custom job bridge class
+  that takes care of receiving Kafka messages and producing/enqueuing ActiveJob
+  jobs)
+* `config.consumer_job_class` (allows configuring a custom job class that
+  processes the enqueued Kafka messages produced by the job bridge)
+* `config.avro_deserializer_class` (allows configuring a custom Apache Avro
+  deserializer class that may implement additional parsing/processing, for
+  example)
+* `config.avro_configure` (allows users to fully customize the
+  `AvroTurf::Messaging` instance)
+* `config.producer_configure` (allows users to fully customize the
+  `WaterDrop::Producer` instance)
+* `config.consumer_configure` (allows users to fully customize the
+  `Karafka::App` instance)
+
+## Producer (WaterDrop)
+
+* No breaking changes, as we wrap it with our Kafka helpers (e.g.
+  `Rimless.message` and the like — their API stayed stable)
+
+## Consumer Setup (Karafka)
+
+<table>
+<tr>
+<td valign="top">
+
+### Karafka 2 (New)
+
+#### Phase 1: Boot
+
+* `bundle exec karafka server`
+* Karafka: load some Karafka (server) defaults
+* Karafka: require `rails` — if available
+* Karafka: require `/app/karafka.rb`
+* `/app/karafka.rb`: require `rimless`
+* `/app/karafka.rb`: `Rimless.consumer.topics(..)`
+* Rimless: `Rimless.consumer -> Rimless::Consumer::App.new` — this configures
+  Karafka (including logging, code reload)
+* Karafka: Karafka server takes over
+  => (set up consumer groups, start listening for messages)
+
+<br><br><br><br><br><br>
+
+#### Phase 2: Message Consumption (batch fetching, single consumption)
+
+* Karafka: receives message(s) on topic (synced by consumer group)
+  => just one Karafka server process handles a single message, per partition
+     (no double processing)
+* Karafka: routes the message(s) of the topic to the Rimless "job bridge"
+  consumer (`Rimless::Consumer::JobBridge`), then all messages of the batch are
+  processed (lazily deserialized) and enqueued as an ActiveJob
+  (`Rimless::Consumer::Job`) — while the decoded message payload is passed as
+  job parameters and serialized/deserialized by ActiveJob (the job execution
+  may then be concurrent via Sidekiq or another ActiveJob adapter)
+  => the Kafka message now leaves the Karafka server process
+
+* ActiveJob: `Rimless::Consumer::Job` is picked up and executed
+  * Rimless: a `Rimless::Consumer::Base` child class is searched by the
+    `consumer` parameter (class inside `/app/consumers`) and instantiated for
+    the job context (hydrating consumer metadata, the message batch containing
+    the single message, etc)
+  * Rimless: `Rimless::Consumer::Base` unpacks the message `event` (e.g.
+    `user_updated`) and dispatches it as a method on the child consumer with
+    the remaining event parameters as arguments
+  * App: `/app/consumers` class kicks in and runs business application logic
+    => e.g. `IdentityApiConsumer.user_updated(user:, **_)`
+
+</td>
+<td valign="top">
+
+### Karafka 1 (Old)
+
+#### Phase 1: Boot
+
+* `bundle exec karafka server`
+* Karafka: load some Karafka (server) defaults
+* Karafka: require `/app/karafka.rb`
+* `/app/karafka.rb`: require `rimless`
+* Rimless: require `railtie` — set up Sidekiq server part
+* `/app/karafka.rb`: `Rimless.consumer.topics(..).boot!`
+* Rimless: `Rimless.consumer -> ConsumerApp.initialize!`
+  * `initialize_rails!`
+  * `initialize_monitors!`
+  * `initialize_karafka!`
+  * `initialize_logger!`
+  * `initialize_code_reload!`
+* Karafka: Karafka server takes over
+  => (set up consumer groups, start listening for messages)
+
+#### Phase 2: Message Consumption (batch fetching, single consumption)
+
+* Karafka: receives message on topic (synced by consumer group)
+  => just one Karafka server process handles a single message, per partition
+     (no double processing)
+* Karafka: run `Rimless::Karafka::PassthroughMapper` for routing (no-op)
+* Karafka: deserialize message payload with `Rimless::Karafka::AvroDeserializer`
+* Karafka: decoded message is passed into `Karafka::Backends::Sidekiq`
+  => (karafka-sidekiq-backend gem)
+* karafka-sidekiq-backend: wrap the message payload with
+  `Rimless::Karafka::Base64Interchanger`
+  => (Ruby object marshalling + base64 encoding for Valkey/Redis transport,
+      to overcome binary encoding issues)
+  => quite high size overhead on Valkey/Redis
+* karafka-sidekiq-backend: enqueue `Rimless::ConsumerJob` with the wrapped
+  message payload
+  => the Kafka message now leaves the Karafka server process
+
+* Sidekiq: `Rimless::ConsumerJob` is picked up and executed
+  * karafka-sidekiq-backend: a `Rimless::BaseConsumer` class is searched for
+    the message (child class inside `/app/consumers`)
+  * Rimless: `Rimless::BaseConsumer` unpacks the message `event` (e.g.
+    `user_updated`) and dispatches it as a method on the child consumer with
+    the remaining event parameters as arguments
+  * App: `/app/consumers` class kicks in and runs business application logic
+    => e.g. `IdentityApiConsumer.user_updated(user:, **_)`
+
+</td>
+</tr>
+</table>
+
+## Testing
+
+See: https://github.com/karafka/karafka-testing/blob/master/2.0-Upgrade.md
+
+* Replace `#karafka_consumer_for` in your specs with `#kafka_consumer_for`
+  (provided and augmented by Rimless to skip job enqueuing and instead
+  perform the wrapped consumer job directly)
+* Replace `#publish_for_karafka` in your specs with `karafka.produce` (in
+  case you did not use the Rimless message producing helpers)

data/doc/upgrade-guide-sources/dep-avro_turf-1.20.md

@@ -0,0 +1,23 @@
+# AvroTurf RubyGem (relevant changes)
+
+* See: https://github.com/dasch/avro_turf/blob/master/CHANGELOG.md
+* Migrated the `avro_turf` gem from `~> 0.11.0` to `~> 1.20`
+
+---
+
+- [Important](#important)
+- [Minor](#minor)
+
+## Important
+
+* The `excon` dependency was upgraded to `>= 0.104, < 2`
+* Removed `sinatra` as a development dependency (our Rimless gem dropped the
+  `sinatra` gem dependency, too)
+* Stopped caching nested sub-schemas
+
+## Minor
+
+* Added compatibility with Avro v1.12.x
+* Added `resolv_resolver` parameter to `AvroTurf::Messaging` to make use of
+  custom domain name resolvers and their options, for example `nameserver` and
+  `timeouts`

data/doc/upgrade-guide-sources/dep-karafka-2.0.md

@@ -0,0 +1,117 @@
+# Karafka RubyGem (relevant changes)
+
+* See: https://github.com/karafka/karafka/blob/master/CHANGELOG.md
+* See: https://github.com/karafka/karafka/wiki/Upgrades-Karafka-2.0
+* Migrated the `karafka` gem from `~> 1.4` to `~> 2.5`
+
+---
+
+- [Important (Structural)](#important-structural)
+- [Important (Logical)](#important-logical)
+- [Important (Configurations)](#important-configurations)
+- [Important (End-user code changes)](#important-end-user-code-changes)
+- [Minor](#minor)
+
+## Important (Structural)
+
+* Removed the topic mappers concept completely
+* Removed support for using sidekiq-backend due to the introduction of
+  multi-threading
+  * Removed the now incompatible `karafka-sidekiq-backend` gem
+  * If you use sidekiq-backend, you have two options:
+    * Leverage Karafka's multi-threading capabilities
+    * Pipe the jobs to Sidekiq yourself (this is what we do with the Rimless
+      gem now)
+* Removed the Responders concept in favor of WaterDrop producer usage
+* Removed all callbacks completely in favor of the finalizer method `#shutdown`
+* Removed single message consumption mode in favor of documentation on how to
+  do it easily yourself (see:
+  https://github.com/karafka/karafka/wiki/Consuming-messages#consuming-messages-one-at-a-time)
+  * In the past, Rimless configured `config.batch_fetching = true` and
+    `config.batch_consuming = false`, resulting in single message processing
+    within the Karafka process, but each Kafka message was enqueued as a
+    Sidekiq worker/job — so message consumption was always concurrent. Batch
+    fetching is now always done by Karafka; adjust `config.max_wait_time` or
+    `config.max_messages` to optimize for latency or throughput (also check
+    `config.kafka[:'enable.partition.eof'] = true`, see:
+    https://karafka.io/docs/Latency-and-Throughput/).
+* Renamed `Karafka::Params::BatchMetadata` to
+  `Karafka::Messages::BatchMetadata`
+* Renamed `Karafka::Params::Params` to `Karafka::Messages::Message`
+* Renamed `#params_batch` in consumers to `#messages` (Rimless adds a
+  compatibility delegation for the old `#params_batch`)
+* Renamed `Karafka::Params::Metadata` to `Karafka::Messages::Metadata`
+* Renamed `Karafka::Fetcher` to `Karafka::Runner` and aligned notification key
+  names
+* Renamed `Karafka::Instrumentation::StdoutListener` to
+  `Karafka::Instrumentation::LoggerListener`
+* Renamed `Karafka::Serializers::JSON::Deserializer` to
+  `Karafka::Deserializers::Payload`
+
+## Important (Logical)
+
+* Changed how the routing style (0.5) behaves. It now builds a single consumer
+  group instead of one per topic (consumer groups: 2.0 uses 1 for all topics,
+  1.4 used 1 per topic)
+* Karafka 2.0 introduces seamless Ruby on Rails integration via `Rails::Railtie`
+  without needing extra configuration (this is reflected in the Rimless gem,
+  as we no longer initialize the Rails application)
+
+## Important (Configurations)
+
+* Karafka 2.0 is powered by librdkafka, Rimless allows configuration via
+  `Rimless.configuration.consumer_configure` and the configuration is split
+  into Karafka settings (root level, see:
+  https://github.com/karafka/karafka/blob/v2.5.5/lib/karafka/setup/config.rb)
+  and Kafka settings (`config.kafka`, see:
+  https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md)
+* Below you can find some of the most significant naming changes in the
+  configuration options:
+  * Root options:
+    * `start_from_beginning` is now `initial_offset` and accepts either
+      'earliest' or 'latest'
+    * `ssl_ca_certs_from_system` is no longer needed, but `kafka`
+      `security.protocol` needs to be set to `ssl`
+    * `batch_fetching` is no longer needed
+    * `batch_consuming` is no longer needed
+    * `serializer` is no longer needed because Responders have been removed
+      from Karafka
+    * `topic_mapper` is no longer needed, as the concept of mapping topic names
+      has been removed from Karafka
+    * `backend` is no longer needed because Karafka is now multi-threaded
+    * `manual_offset_management` now needs to be set on a per-topic basis
+  * Kafka options:
+    * `kafka.seed_brokers` is now `bootstrap.servers` without the protocol
+      definition
+    * `kafka.heartbeat_interval` is no longer needed.
+    * SASL and SSL options changes are described in their own section.
+
+## Important (End-user code changes)
+
+* Remove WaterDrop setup code from your `karafka.rb`:
+```ruby
+# This can be safely removed
+monitor.subscribe('app.initialized') do
+  WaterDrop.setup { |config| config.deliver = !Karafka.env.test? }
+end
+```
+* Remove direct WaterDrop listener references from your `karafka.rb`:
+```ruby
+# This can be safely removed
+Karafka.monitor.subscribe(WaterDrop::Instrumentation::LoggerListener.new)
+```
+* Remove the `KarafkaApp.boot!` from the end of `karafka.rb`:
+```ruby
+# Remove this
+KarafkaApp.boot!
+# or in case of Rimless:
+Rimless.consumer.topics(...).boot! # just the `.boot!` call
+```
+
+## Minor
+
+* No `dry-*` gems are used as dependencies anymore
+* Added `KARAFKA_REQUIRE_RAILS` to disable the default Rails require, to run
+  Karafka without Rails despite having Rails in the Gemfile
+* Allow running boot-file-less Rails setup Karafka CLI commands where
+  configuration is done in initializers

data/doc/upgrade-guide-sources/dep-waterdrop-2.8.md

@@ -0,0 +1,30 @@
+# WaterDrop RubyGem (relevant changes)
+
+* See: https://github.com/karafka/waterdrop/blob/master/CHANGELOG.md
+* Migrated the `waterdrop` gem from `~> 1.4` to `~> 2.8`
+
+---
+
+- [Important](#important)
+- [Minor](#minor)
+
+## Important
+
+* Replaced `ruby-kafka` with `rdkafka` (karafka-rdkafka, native gem/lib)
+* The new underlying Kafka library has different/renamed options (see:
+  https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md)
+* No `dry-*` gems are used as dependencies anymore
+* Complete redesign of the API (this is wrapped by Rimless helpers, which
+  remained stable)
+* All time-related values are now configured in milliseconds instead of some
+  being in seconds and some in milliseconds
+
+## Minor
+
+* Added support for sending tombstone messages
+* Changed auto-generated ID from `SecureRandom#uuid` to `SecureRandom#hex(6)`
+* Introduced transactions support
+* Added support for producing messages with arrays of strings in headers
+  (KIP-82)
+* Added `WaterDrop::ConnectionPool` for efficient connection pooling using the
+  proven `connection_pool` gem

data/gemfiles/rails_8.0.gemfile

@@ -7,7 +7,6 @@ gem "bundler", ">= 2.6", "< 5"
 gem "countless", "~> 2.2"
 gem "factory_bot", "~> 6.2"
 gem "guard-rspec", "~> 4.7"
-gem "railties", "~> 8.0.0"
 gem "rake", "~> 13.0"
 gem "redcarpet", "~> 3.5"
 gem "rspec", "~> 3.12"
@@ -19,6 +18,7 @@ gem "timecop", ">= 0.9.6"
 gem "vcr", "~> 6.0"
 gem "yard", ">= 0.9.28"
 gem "yard-activesupport-concern", ">= 0.0.1"
+gem "activejob", "~> 8.0.0"
 gem "activesupport", "~> 8.0.0"
 
 gemspec path: "../"

data/gemfiles/rails_8.1.gemfile

@@ -7,7 +7,6 @@ gem "bundler", ">= 2.6", "< 5"
 gem "countless", "~> 2.2"
 gem "factory_bot", "~> 6.2"
 gem "guard-rspec", "~> 4.7"
-gem "railties", "~> 8.1.0"
 gem "rake", "~> 13.0"
 gem "redcarpet", "~> 3.5"
 gem "rspec", "~> 3.12"
@@ -19,6 +18,7 @@ gem "timecop", ">= 0.9.6"
 gem "vcr", "~> 6.0"
 gem "yard", ">= 0.9.28"
 gem "yard-activesupport-concern", ">= 0.0.1"
+gem "activejob", "~> 8.1.0"
 gem "activesupport", "~> 8.1.0"
 
 gemspec path: "../"

data/lib/rimless/configuration.rb

@@ -71,7 +71,18 @@ module Rimless
 
     # At least one broker of the Apache Kafka cluster
     config_accessor(:kafka_brokers) do
-      ENV.fetch('KAFKA_BROKERS', 'kafka://message-bus.local:9092').split(',')
+      ENV.fetch('KAFKA_BROKERS', 'kafka://message-bus.local:9092')
+         .split(',').map { |uri| uri.split('://', 2).last }.join(',')
+    end
+
+    # A custom writer for the kafka brokers configuration.
+    #
+    # @param val [String, Array<String>] the new kafka brokers list
+    def kafka_brokers=(val)
+      self[:kafka_brokers] =
+        Array(val).join(',').split(',')
+                  .map { |uri| uri.split('://', 2).last }
+                  .join(',')
     end
 
     # The source Apache Avro schema files location (templates)
@@ -96,9 +107,39 @@ module Rimless
                 'http://schema-registry.message-bus.local')
     end
 
-    # The Sidekiq job queue to use for consuming jobs
+    # This configuration allows users to configure a customized logger listener
+    # (which is bound to +Rimless.logger+). When configured to a falsy value
+    # (eg. +false+, or +nil+), no listener is installed by Rimless to Karafka.
+    config_accessor(:consumer_logger_listener) do
+      Karafka::Instrumentation::LoggerListener.new(
+        log_polling: false
+      )
+    end
+
+    # This setting allows users to configure a custom job bridge class, which
+    # takes care of receiving Kafka messages and produce/enqueue ActiveJob
+    # jobs. The configured class must be +Karafka::BaseConsumer+ compatible,
+    # for Karafka.
+    config_accessor(:job_bridge_class) { Rimless::Consumer::JobBridge }
+
+    # This configuration allows to choose a different consumer job class,
+    # enqueued by the +job_bridge_class+. This allows fully customized handling
+    # on user applications.
+    config_accessor(:consumer_job_class) { Rimless::Consumer::Job }
+
+    # This configuration allows to choose the default Apache Avro deserializer
+    # class, which is used by the Karafka consumer while using the
+    # +Rimless.consumer.topics+ helper.
+    config_accessor(:avro_deserializer_class) do
+      Rimless::Consumer::AvroDeserializer
+    end
+
+    # The ActiveJob job queue to use for consuming jobs
     config_accessor(:consumer_job_queue) do
-      ENV.fetch('KAFKA_SIDEKIQ_JOB_QUEUE', 'default').to_sym
+      ENV.fetch(
+        'KAFKA_JOB_QUEUE',
+        ENV.fetch('KAFKA_SIDEKIQ_JOB_QUEUE', 'default')
+      ).to_sym
     end
 
     # A custom writer for the consumer job queue name.
@@ -106,10 +147,43 @@ module Rimless
     # @param val [String, Symbol] the new job queue name
     def consumer_job_queue=(val)
       self[:consumer_job_queue] = val.to_sym
+
       # Refresh the consumer job queue
-      Rimless::ConsumerJob.sidekiq_options(
-        queue: Rimless.configuration.consumer_job_queue
-      )
+      consumer_job_class.queue_as(val)
     end
+
+    # This configuration block allows users to fully customized the
+    # +AvroTurf::Messaging+ instance. The Rimless default parameters hash is
+    # injected as argument to the configured block. The result of the block is
+    # then used to instantiate +AvroTurf::Messaging+, use like this.
+    #
+    #   ->(config) { config.merge(connect_timeout: 5) }
+    #
+    # See: https://bit.ly/4r0mDnw
+    config_accessor(:avro_configure) { ->(config) { config } }
+
+    # This configuration block allows users to fully customize the
+    # +WaterDrop::Producer+ instance (accessible as +Rimless.producer+). The
+    # Rimless settings are already applied when the block is called. The
+    # +WaterDrop::Config+ is then injected as argument to the given block, and
+    # can be used regular like this:
+    #
+    #   ->(config) { config.kafka[:'request.required.acks'] = -1 }
+    #
+    # See: https://bit.ly/4r5Uprv (+config.*+ root level WaterDrop settings)
+    # See: https://bit.ly/3OtIfeu (+config.kafka+ settings)
+    config_accessor(:producer_configure) { ->(config) { config } }
+
+    # This configuration block allows users to fully customize the
+    # +Karafka::App+ instance (accessible as +Rimless.consumer+). The Rimless
+    # settings are already applied when the block is called. The
+    # +Karafka::Setup::Config+ is then injected as argument to the given block,
+    # and can be used regular like this:
+    #
+    #   ->(config) { config.kafka[:'request.required.acks'] = -1 }
+    #
+    # See: https://bit.ly/3MAF6Jk (+config.*+ root level Karafka settings)
+    # See: https://bit.ly/3OtIfeu (+config.kafka+ settings)
+    config_accessor(:consumer_configure) { ->(config) { config } }
   end
 end