rimless 2.9.0 → 3.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '0458c4cf20641ca5f2062e98d61d1b01a4160639a323bf866a9dbc20fdb318de'
-  data.tar.gz: ba7ad3a77a0a23b1c32d23f77fb9464bb95de92177b44f4b726e12511f5aa127
+  metadata.gz: a84c6d640596ee56269220afae67746c855c1fc06c91819312c6a0a9626ed422
+  data.tar.gz: 7179ae84177669855f9e38b2d044ccf35ccb3405f42b94aa3275ce75907b0cea
 SHA512:
-  metadata.gz: 5125ef4597b4fff7bf7d47a0ca6c4093998cb34dea99387eab4cbd4eb04656547be0f52fbadb6d488a4f97dcf4d0056fdf8097f3cec38031500b65bfe6af3ac0
-  data.tar.gz: b6971835235fb6ecac1435e78d27c97d5534a2f28df60176457860bc6300e280a7420c63780b482550963af519b0a0299a0c9b0f5edbf46f64d3d92d2d3a747e
+  metadata.gz: 11a1fdcf0a778e581c36055cf4986880940d0fe5b41ec30b569598ad301ce51aae4cf9e2278983d33c77c0e93209b0224021d3b3ba138ac288ce83eb87a30707
+  data.tar.gz: c5d12b9ca331ea5b1000576c06291ecb1f8d336f2cd25f13ade6c2b509b11a83e50f1f23be3077399c70ec2356623ccfe45ee6edba0177ddeef6378e6805a6d2

data/Appraisals

@@ -1,11 +1,11 @@
 # frozen_string_literal: true
 
 appraise 'rails-8.0' do
+  gem 'activejob', '~> 8.0.0'
   gem 'activesupport', '~> 8.0.0'
-  gem 'railties', '~> 8.0.0'
 end
 
 appraise 'rails-8.1' do
+  gem 'activejob', '~> 8.1.0'
   gem 'activesupport', '~> 8.1.0'
-  gem 'railties', '~> 8.1.0'
 end

data/CHANGELOG.md

@@ -2,6 +2,72 @@
 
 * TODO: Replace this bullet point with an actual description of a change.
 
+### 3.0.0 (1 April 2026)
+
+Breaking changes: ([#73](https://github.com/hausgold/rimless/pull/73))
+
+* Upgraded `karafka` from `~> 1.4` to `~> 2.5` — now powered by librdkafka
+  instead of ruby-kafka. Consumer groups are now shared (one for all topics
+  instead of one per topic). Batch fetching is always on; tune
+  `max_wait_time`/`max_messages` for latency vs throughput.
+  See `UPGRADING.md` for details.
+
+* Upgraded `waterdrop` from `~> 1.4` to `~> 2.8` — all time-related config
+  values are now in milliseconds. The Rimless producer helpers (`Rimless.message`
+  etc.) remain unchanged.
+
+* Upgraded `avro_turf` from `~> 0.11.0` to `~> 1.20`
+
+* Switched from Sidekiq to ActiveJob for consumer job processing. The
+  `karafka-sidekiq-backend` gem is no longer used.
+
+* Renamed constants (old → new):
+  - `Rimless::BaseConsumer` → `Rimless::Consumer::Base`
+  - `Rimless::ConsumerApp` → `Rimless::Consumer::App`
+  - `Rimless::ConsumerJob` → `Rimless::Consumer::Job`
+  - `Rimless::Karafka::AvroDeserializer` → `Rimless::Consumer::AvroDeserializer`
+
+* Removed constants:
+  - `Rimless::Karafka::Base64Interchanger` (no longer needed)
+  - `Rimless::Karafka::PassthroughMapper` (mappers removed in Karafka 2)
+
+* Anonymous consumer classes are no longer supported. Because Karafka is no
+  longer loaded within the ActiveJob worker process, consumer class names must
+  be serializable (via `constantize`). Rewrite anonymous consumers as named
+  classes. See `UPGRADING.md` for details.
+
+* Each Kafka message in a batch is now individually marked as consumed after
+  being enqueued to ActiveJob (`mark_as_consumed`). This may differ from
+  previous `karafka-sidekiq-backend` behavior depending on your Karafka
+  offset management configuration. See `UPGRADING.md` for details.
+
+* `KAFKA_BROKERS` format changed — protocol prefix (`kafka://`) is no longer
+  required. Use plain `host:port` CSV (e.g. `your.domain:9092,host:port`).
+  The old `kafka://` format is still accepted for backwards compatibility.
+
+* Consumer boot flow changed:
+  - Remove `.boot!` from the end of `Rimless.consumer.topics(...)` in
+    `karafka.rb`
+  - Remove `WaterDrop.setup` and `WaterDrop::Instrumentation::LoggerListener`
+    subscriptions from `karafka.rb`
+
+New configuration options:
+
+* `config.job_bridge_class` — custom job bridge for Kafka → ActiveJob
+* `config.consumer_job_class` — custom consumer job class
+* `config.consumer_logger_listener` — customize Karafka logging listener
+* `config.avro_deserializer_class` — custom Apache Avro deserializer
+* `config.avro_configure` — fully customize the `AvroTurf::Messaging` instance
+* `config.producer_configure` — fully customize the `WaterDrop::Producer` instance
+* `config.consumer_configure` — fully customize the `Karafka::App` instance
+
+Testing changes:
+
+* Replace `#karafka_consumer_for` with `#kafka_consumer_for` in specs
+* Replace `#publish_for_karafka` with `karafka.produce` in specs
+
+See [`UPGRADING.md`](./UPGRADING.md) for a complete migration guide.
+
 ### 2.9.0 (18 February 2026)
 
 * Dropped 3rd-level gem dependencies which are not directly used

data/Gemfile

@@ -13,7 +13,6 @@ gem 'bundler', '>= 2.6', '< 5'
 gem 'countless', '~> 2.2'
 gem 'factory_bot', '~> 6.2'
 gem 'guard-rspec', '~> 4.7'
-gem 'railties', '>= 8.0'
 gem 'rake', '~> 13.0'
 gem 'redcarpet', '~> 3.5'
 gem 'rspec', '~> 3.12'

data/README.md

@@ -6,13 +6,15 @@
 [![Test Ratio](https://automate-api.hausgold.de/v1/coverage_reports/rimless/ratio.svg)](https://knowledge.hausgold.de/coverage)
 [![API docs](https://automate-api.hausgold.de/v1/coverage_reports/rimless/documentation.svg)](https://www.rubydoc.info/gems/rimless)
 
-This project is dedicated to ship a ready to use [Apache
+This project is dedicated to shipping a ready-to-use [Apache
 Kafka](https://kafka.apache.org/) / [Confluent Schema
 Registry](https://docs.confluent.io/current/schema-registry/index.html) /
-[Apache Avro](https://avro.apache.org/) message producing toolset by making use
-of the [WaterDrop](https://rubygems.org/gems/waterdrop) and
+[Apache Avro](https://avro.apache.org/) message streaming toolset by making use
+of the [WaterDrop](https://rubygems.org/gems/waterdrop),
+[Karafka](https://rubygems.org/gems/karafka) and
 [AvroTurf](https://rubygems.org/gems/avro_turf) gems. It comes as an
-opinionated framework which sets up solid conventions for producing messages.
+opinionated framework that sets up solid conventions for producing and
+consuming messages.
 
 - [Installation](#installation)
 - [Usage](#usage)
@@ -60,7 +62,7 @@ $ gem install rimless
 
 ### Configuration
 
-You can configure the rimless gem via an Rails initializer, by environment
+You can configure the rimless gem via a Rails initializer, by environment
 variables or on demand. Here we show a common Rails initializer example:
 
 ```ruby
@@ -82,33 +84,42 @@ Rimless.configure do |conf|
 
   # The list of Apache Kafka brokers for cluster discovery,
   # set to HAUSGOLD defaults when not set
-  conf.kafka_brokers = 'kafka://your.domain:9092,kafka..'
+  conf.kafka_brokers = 'kafka://your.domain:9092,kafka..' # old format
+  conf.kafka_brokers = 'your.domain:9092,host:port..' # new format
 
   # The Confluent Schema Registry API URL,
   # set to HAUSGOLD defaults when not set
   conf.schema_registry_url = 'http://your.schema-registry.local'
 
-  # The Sidekiq job queue to use for consuming jobs
+  # The ActiveJob queue to use for consuming jobs
   config.consumer_job_queue = 'default'
 end
 ```
 
-The rimless gem comes with sensitive defaults as you can see. For most users an
-extra configuration is not needed.
+The rimless gem comes with sensible defaults as you can see. For most users an
+extra configuration is not needed. For more details and all available
+configurations, check the
+[`lib/rimless/configuration.rb`](lib/rimless/configuration.rb) file.
 
 #### Available environment variables
 
-The rimless gem can be configured hardly with its configuration code block like
+The rimless gem can also be configured with its configuration code block as
 shown before. Respecting the [twelve-factor app](https://12factor.net/)
 concerns, the gem allows you to set almost all configurations (just the
 relevant ones for runtime) via environment variables. Here comes a list of
 available configuration options:
 
-* **KAFKA_ENV**: The application environment. Falls back to `Rails.env` when available.
-* **KAFKA_CLIENT_ID**: The Apache Kafka client identifier, falls back the the local application name.
-* **KAFKA_BROKERS**: A comma separated list of Apache Kafka brokers for cluster discovery (Plaintext, no-auth/no-SSL only for now) (eg. `kafka://your.domain:9092,kafka..`)
-* **KAFKA_SCHEMA_REGISTRY_URL**: The Confluent Schema Registry API URL to use for schema registrations.
-* **KAFKA_SIDEKIQ_JOB_QUEUE**: The Sidekiq job queue to use for consuming jobs. Falls back to `default`.
+* **KAFKA_ENV**: The application environment. Falls back to `Rails.env` when
+  available.
+* **KAFKA_CLIENT_ID**: The Apache Kafka client identifier, falls back to the
+  local application name.
+* **KAFKA_BROKERS**: A comma-separated list of Apache Kafka brokers for cluster
+  discovery (Plaintext, no-auth/no-SSL by default) (eg.
+  `your.domain:9092,host:port..`)
+* **KAFKA_SCHEMA_REGISTRY_URL**: The Confluent Schema Registry API URL to use
+  for schema registrations.
+* **KAFKA_JOB_QUEUE**: The ActiveJob queue to use for consuming jobs.
+  Falls back to `default`. (falls back to `KAFKA_SIDEKIQ_JOB_QUEUE`)
 
 ### Conventions
 
@@ -158,13 +169,13 @@ templates](https://ruby-doc.org/stdlib-2.6.2/libdoc/erb/rdoc/ERB.html) and
 painless JSON validation of them.
 
 First things first, by convention the rimless gem looks for Apache Avro schema
-ERB templates on the `$(pwd)/config/avro_schemas` directory. Nothing special
+ERB templates in the `$(pwd)/config/avro_schemas` directory. Nothing special
 from the Rails perspective. You can also reconfigure the file locations, just
 [see the configuration
 block](https://github.com/hausgold/rimless/blob/master/lib/rimless/configuration.rb#L36).
 
 Each schema template MUST end with the `.avsc.erb` extension to be picked up,
-even in recursive directory structures.  You can make use of the ERB templating
+even in recursive directory structures. You can make use of the ERB templating
 or not, but rimless just looks for these templates. When it comes to
 structuring the Avro Schemas it is important that the file path reflects the
 embedded schema namespace correctly. So when `$(pwd)/config/avro_schemas` is our
@@ -178,7 +189,7 @@ The corresponding Avro Schema template is located at
 be fancy. The automatic schema compiler picks up the dynamically/runtime set
 namespace from the schema definition and converts it to its respective
 directory structure. So when you boot your application container/instance
-inside your *canary*  environment, the schemas/messages should reflect this so
+inside your *canary* environment, the schemas/messages should reflect this so
 they do not mix with other environments.
 
 Example time. **$(pwd)/config/avro_schemas/identity_api/user_v1.avsc.erb**:
@@ -237,7 +248,7 @@ The compiled Avro Schemas are written to the
 `$(pwd)/config/avro_schemas/compiled/` directory by default. You can
 [reconfigure the
 location](https://github.com/hausgold/rimless/blob/master/lib/rimless/configuration.rb#L44)
-if needed. For VCS systems like Git it is useful to create an relative ignore
+if needed. For VCS systems like Git it is useful to create a relative ignore
 list at `$(pwd)/config/avro_schemas/.gitignore` with the following contents:
 
 ```gitignore
@@ -248,8 +259,8 @@ compiled/
 
 Under the hood the rimless gem makes use of the [WaterDrop
 gem](https://rubygems.org/gems/waterdrop) to send messages to the Apache Kafka
-cluster. But with the addition to send Apache Avro encoded messages with a
-single call. Here comes some examples how to use it:
+cluster, but with the added ability to send Apache Avro encoded messages with a
+single call. Here are some examples of how to use it:
 
 ```ruby
 metadata = { hobbies: %w(dancing singing sports) }
@@ -279,7 +290,7 @@ Rimless.raw_message(data: encoded, topic: :users)
 
 # In case you want to send messages to a non-local application topic you can
 # specify the application, too. This allows you to send a message to the
-# +<ENV>.address-api.addresses+ from you local identity-api.
+# +<ENV>.address-api.addresses+ from your local identity-api.
 Rimless.raw_message(data: encoded, topic: { name: :users, app: 'address-api' })
 # Also works with the Apache Avro encoding variant
 Rimless.message(data: user, schema: :user_v1,
@@ -295,8 +306,8 @@ Rimless.async_raw_message(data: encoded, topic: :users)
 The rimless gem makes it super easy to build consumer logic right into your
 (Rails, standalone) application by utilizing the [Karafka
 framework](https://github.com/karafka/karafka) under the hood. When you have
-the rimless gem already installed you are ready to rumble to setup your
-application to consume Apache Kafka messages. Just run the `$ rake
+the rimless gem already installed, you are ready to set up your application to
+consume Apache Kafka messages. Just run the `$ rake
 rimless:install` and all the consuming setup is done for you.
 
 Afterwards you find the `karafka.rb` file at the root of your project together
@@ -312,7 +323,7 @@ architecture looks like this:
               v
   +-----------------------------+
   | Karafka/Rimless Consumer    |    +--------------------------------------+
-  |   Shares a single consumer  |--->| Sidekiq                              |
+  |   Shares a single consumer  |--->| ActiveJob                            |
   |   group, multiple processes |    |   Runs the consumer class (children  |
   +-----------------------------+    |   of Rimless::BaseConsumer) for each |
                                      |   message (Rimless::ConsumerJob),    |
@@ -321,15 +332,15 @@ architecture looks like this:
 ```
 
 This architecture allows the consumer process to run mostly non-blocking and
-the messages can be processed concurrently via Sidekiq. (including the error
+the messages can be processed concurrently via ActiveJob. (including the error
 handling and retrying)
 
 #### Routing messages to consumers
 
 The `karafka.rb` file at the root of your project is dedicated to configure the
 consumer process, including the routing table. The routing is as easy as it
-gets by following this pattern: `topic => consumer`. Here comes a the full
-examples:
+gets by following this pattern: `topic => consumer`. Here is the full
+example:
 
 ```ruby
 # Setup the topic-consumer routing table and boot the consumer application
@@ -338,7 +349,7 @@ Rimless.consumer.topics(
 ).boot!
 ```
 
-The key side of the hash is anything which is understood by the `Rimless.topic`
+The key side of the hash is anything that is understood by the `Rimless.topic`
 method. With one addition: you can change `:name` to `:names` and pass an array
 of strings or symbols to listen to multiple application topics with a single
 configuration line.
@@ -360,9 +371,9 @@ Rimless.consumer.topics(
 #### Consuming event messages
 
 By convention it makes sense to produce messages with various event types on a
-single Apache Kafka topic. This is fine, they just must follow a single
-constrain: each message must contain an `event`-named field at the Apache Avro
-schema with a dedicated name. This allow to structure data at Kafka like this:
+single Apache Kafka topic. This is fine; they just must follow a single
+constraint: each message must contain an `event`-named field in the Apache Avro
+schema with a dedicated name. This allows structuring data in Kafka like this:
 
 ```
 Topic: production.users-api.users
@@ -370,7 +381,7 @@ Events: user_created, user_updated, user_deleted
 ```
 
 While respecting this convention your consumer classes will be super clean. See
-the following example: (we keep the users api example)
+the following example: (we keep the Users API example)
 
 ```ruby
 class UserApiConsumer < ApplicationConsumer
@@ -385,7 +396,7 @@ schema fields of it, except the event field. The messages will be automatically
 decoded with the help of the schema registry. All hashes/arrays ship deeply
 symbolized keys for easy access.
 
-**Heads up!** All messages with events which are not reflected by a method will
+**Heads up!** All messages with events that are not reflected by a method will
 just be ignored.
 
 See the automatically generated spec (`spec/consumers/custom_consumer_spec.rb`)
@@ -408,20 +419,11 @@ $ rake rimless:routes
 #### Starting the consumer process(es)
 
 From system integration perspective you just need to start the consumer
-processes and Sidekiq to get the thing going. Rimless allows you to start the
+processes and ActiveJob to get the thing going. Rimless allows you to start the
 consumer with `$ rake rimless:consumer` or you can just use the [Karafka
 binary](https://github.com/karafka/karafka/wiki/Fetching-messages) to start the
 consumer (`$ bundle exec karafka server`). Both work identically.
 
-When running inside a Rails application the consumer application initialization
-is automatically done for Sidekiq. Otherwise you need to initialize the
-consumer application manually with:
-
-```ruby
-# Manual consumer application initialization
-Sidekiq.configure_server { Rimless.consumer.initialize! }
-```
-
 ### Encoding/Decoding messages
 
 By convention we focus on the [Apache Avro](https://avro.apache.org/) data
@@ -430,14 +432,14 @@ gem and the rimless gem adds some neat helpers on top of it. Here are a few
 examples to show how rimless can be used to encode/decode Apache Avro data:
 
 ```ruby
-# Encode a data structure (no matter of symbolized, or stringified keys, or
+# Encode a data structure (regardless of symbolized or stringified keys, or
 # non-simple types) to Apache Avro format
 encoded = Rimless.encode(user, schema: 'user_v1')
 
 # Works the same for symbolized schema names
 encoded = Rimless.encode(user, schema: :user_v1)
 
-# Also supports the resolution of deep relative schemes
+# Also supports the resolution of deep relative schemas
 # (+.user.address+ becomes +<ENV>.<APP>.user.address+)
 encoded = Rimless.encode(user.address, schema: '.user.address')
 
@@ -448,15 +450,15 @@ decoded = Rimless.decode('your-avro-binary-data-here')
 
 #### Handling of schemaless deep blobs
 
-Apache Avro is by design a strict, type casted format which does not allow
+Apache Avro is by design a strict, type-cast format that does not allow
 undefined mix and matching of deep structures. This is fine because it forces
 the producer to think twice about the schema definition. But sometimes there is
 unstructured data inside of entities. Think of a metadata hash on a user entity
-were the user (eg. a frontend client) just can add whatever comes to his mind
-for later processing. Its not searchable, its never touched by the backend, but
-its present.
+where the user (e.g. a frontend client) can just add whatever comes to mind
+for later processing. It's not searchable, it's never touched by the backend,
+but it's present.
 
-That's a case we're experienced and kind of solved on the rimless gem. You can
+That's a case we've experienced and kind of solved in the rimless gem. You can
 make use of the `Rimless.avro_schemaless_h` method to [sparsify the data
 recursively](https://github.com/simplymeasured/sparsify). Say you have the
 following metadata hash:
@@ -475,7 +477,7 @@ metadata = {
 ```
 
 It's messy, by design. From the Apache Avro perspective you just can define a
-map. The map keys are assumed to be strings - and the most hitting value data
+map. The map keys are assumed to be strings — and the most fitting value data
 type is a string, too. That's where hash sparsification comes in. The resulting
 metadata hash looks like this and can be encoded by Apache Avro:
 
@@ -492,18 +494,18 @@ Rimless.avro_schemaless_h(metadata)
 ```
 
 With the help of the [sparsify gem](https://rubygems.org/gems/sparsify) you can
-also revert this to its original form. But with the loss of data type
-correctness. Another approach can be used for these kind of scenarios: encoding
+also revert this to its original form, but with the loss of data type
+correctness. Another approach can be used for these kinds of scenarios: encoding
 the schemaless data with JSON and just set the metadata field on the Apache
 Avro schema to be a string. Choice is yours.
 
 ### Writing tests for your messages
 
-Producing messages is a bliss with the rimless gem, but producing code needs to
+Producing messages is a breeze with the rimless gem, but producing code needs to
 be tested as well. That's why the gem ships some RSpec helpers and matchers for
 this purpose. A common situation is also handled by the RSpec extension: on the
-test environment (eg. a continuous integration service) its not likely to have
-a Apache Kafka/Confluent Schema Registry cluster available. That's why actual
+test environment (e.g. a continuous integration service) it's not likely to have
+an Apache Kafka/Confluent Schema Registry cluster available. That's why actual
 calls to Kafka/Schema Registry are mocked away.
 
 First of all, just add `require 'rimless/rspec'` to your `spec_helper.rb` or
@@ -525,8 +527,8 @@ end
 
 Nothing special, not really fancy. A more complex situation occurs when you
 separate your Kafka message producing logic inside an asynchronous job (eg.
-Sidekiq or ActiveJob). Therefore is the `have_sent_kafka_message` matcher
-available. Example time:
+ActiveJob). The `have_sent_kafka_message` matcher is available for
+this purpose. Example time:
 
 ```ruby
 describe 'message producer job' do
@@ -559,7 +561,7 @@ describe 'message producer job' do
     expect { action }.to \
       have_sent_kafka_message.with(key: String, topic: anything)
     #                 mind the order --^
-    #                 its a argument list validation, all keys must be named
+    #                 it's an argument list validation, all keys must be named
   end
 
   it 'sends the correct user data' do
@@ -571,14 +573,14 @@ describe 'message producer job' do
     expect { nil }.not_to have_sent_kafka_message
   end
 
-  it 'allows complex expactations' do
+  it 'allows complex expectations' do
     expect { action; action }.to \
       have_sent_kafka_message('test.identity_api.user_v1')
         .with(key: user.id, topic: 'test.identity-api.users').twice
         .with_data(firstname: 'John', lastname: 'Doe').twice
   end
 
-  it 'allows to capture messages to check them in detail' do
+  it 'allows capturing messages to check them in detail' do
     (capture_kafka_messages { action }).tap do |messages|
       expect(messages.first[:data]).to \
         match(a_hash_including('entity' => a_hash_including('items' => items)))

data/Rakefile

@@ -15,10 +15,19 @@ Countless.configure do |config|
       pattern: %r{/lib(/rimless)?/[^/]+\.rb$} },
     { name: 'Top-levels specs', test: true, dir: 'spec',
       pattern: %r{/spec(/rimless)?/[^/]+_spec\.rb$} },
-    { name: 'RSpec matchers', pattern: 'lib/rimless/rspec/**/*.rb' },
-    { name: 'RSpec matchers specs', test: true,
+    { name: 'Initializers', pattern: 'lib/rimless/initializers/**/*.rb' },
+    { name: 'Compatibilities', pattern: 'lib/rimless/compatibility/**/*.rb' },
+    { name: 'Compatibilities specs', test: true,
+      pattern: 'spec/rimless/compatibility/**/*_spec.rb' },
+    { name: 'Consumer', pattern: 'lib/rimless/consumer/**/*.rb' },
+    { name: 'Consumer specs', test: true,
+      pattern: 'spec/rimless/consumer/**/*_spec.rb' },
+    { name: 'Extensions', pattern: 'lib/rimless/extensions/**/*.rb' },
+    { name: 'Extensions specs', test: true,
+      pattern: 'spec/rimless/extensions/**/*_spec.rb' },
+    { name: 'RSpec extensions', pattern: 'lib/rimless/rspec/**/*.rb' },
+    { name: 'RSpec extensions specs', test: true,
       pattern: 'spec/rimless/rspec/**/*_spec.rb' },
-    { name: 'Rake Tasks', pattern: 'lib/rimless/tasks/**/*' },
-    { name: 'Karafka Extensions', pattern: 'lib/rimless/karafka/**/*.rb' }
+    { name: 'Rake Tasks', pattern: 'lib/rimless/tasks/**/*' }
   ]
 end