rdkafka 0.25.1-aarch64-linux-musl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (60) hide show
  1. checksums.yaml +7 -0
  2. data/CHANGELOG.md +300 -0
  3. data/Gemfile +13 -0
  4. data/Gemfile.lint +14 -0
  5. data/Gemfile.lint.lock +123 -0
  6. data/MIT-LICENSE +22 -0
  7. data/README.md +181 -0
  8. data/Rakefile +96 -0
  9. data/bin/verify_kafka_warnings +37 -0
  10. data/docker-compose-ssl.yml +35 -0
  11. data/docker-compose.yml +25 -0
  12. data/ext/librdkafka.so +0 -0
  13. data/lib/rdkafka/abstract_handle.rb +134 -0
  14. data/lib/rdkafka/admin/acl_binding_result.rb +51 -0
  15. data/lib/rdkafka/admin/config_binding_result.rb +30 -0
  16. data/lib/rdkafka/admin/config_resource_binding_result.rb +19 -0
  17. data/lib/rdkafka/admin/create_acl_handle.rb +31 -0
  18. data/lib/rdkafka/admin/create_acl_report.rb +25 -0
  19. data/lib/rdkafka/admin/create_partitions_handle.rb +30 -0
  20. data/lib/rdkafka/admin/create_partitions_report.rb +7 -0
  21. data/lib/rdkafka/admin/create_topic_handle.rb +32 -0
  22. data/lib/rdkafka/admin/create_topic_report.rb +27 -0
  23. data/lib/rdkafka/admin/delete_acl_handle.rb +33 -0
  24. data/lib/rdkafka/admin/delete_acl_report.rb +25 -0
  25. data/lib/rdkafka/admin/delete_groups_handle.rb +33 -0
  26. data/lib/rdkafka/admin/delete_groups_report.rb +27 -0
  27. data/lib/rdkafka/admin/delete_topic_handle.rb +32 -0
  28. data/lib/rdkafka/admin/delete_topic_report.rb +27 -0
  29. data/lib/rdkafka/admin/describe_acl_handle.rb +33 -0
  30. data/lib/rdkafka/admin/describe_acl_report.rb +26 -0
  31. data/lib/rdkafka/admin/describe_configs_handle.rb +36 -0
  32. data/lib/rdkafka/admin/describe_configs_report.rb +60 -0
  33. data/lib/rdkafka/admin/incremental_alter_configs_handle.rb +36 -0
  34. data/lib/rdkafka/admin/incremental_alter_configs_report.rb +60 -0
  35. data/lib/rdkafka/admin.rb +894 -0
  36. data/lib/rdkafka/bindings.rb +626 -0
  37. data/lib/rdkafka/callbacks.rb +475 -0
  38. data/lib/rdkafka/config.rb +407 -0
  39. data/lib/rdkafka/consumer/headers.rb +80 -0
  40. data/lib/rdkafka/consumer/message.rb +87 -0
  41. data/lib/rdkafka/consumer/partition.rb +55 -0
  42. data/lib/rdkafka/consumer/topic_partition_list.rb +173 -0
  43. data/lib/rdkafka/consumer.rb +858 -0
  44. data/lib/rdkafka/defaults.rb +106 -0
  45. data/lib/rdkafka/error.rb +116 -0
  46. data/lib/rdkafka/helpers/oauth.rb +90 -0
  47. data/lib/rdkafka/helpers/time.rb +19 -0
  48. data/lib/rdkafka/metadata.rb +139 -0
  49. data/lib/rdkafka/native_kafka.rb +224 -0
  50. data/lib/rdkafka/producer/delivery_handle.rb +40 -0
  51. data/lib/rdkafka/producer/delivery_report.rb +50 -0
  52. data/lib/rdkafka/producer/partitions_count_cache.rb +226 -0
  53. data/lib/rdkafka/producer.rb +521 -0
  54. data/lib/rdkafka/version.rb +10 -0
  55. data/lib/rdkafka.rb +55 -0
  56. data/package-lock.json +331 -0
  57. data/package.json +9 -0
  58. data/rdkafka.gemspec +84 -0
  59. data/renovate.json +94 -0
  60. metadata +177 -0
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 66d7ba894eadbe3db2aa851eadb7d4d0382eec9ad8d9fc62f5071fe72e058a56
4
+ data.tar.gz: '0478c305f812a6c75990957907e51732d7c31aca85c49e808145f61199f56ef8'
5
+ SHA512:
6
+ metadata.gz: a0290fc85b2075a3b59d39049a13614ccbbaf0cc9d10fffb0e99fe19357bd0e83269bca44ad39efbecd6ff27fe97e11878992371c4a12557b08dd78c996bc9bb
7
+ data.tar.gz: 4dc76ed4fb747b3ad3bab278d80eb722081be41200552f0c646e50e4fd33bb09f25dbc937fcf41dd37f83c613b3e7273b37384f0e610a58373de2b6b4b9c46be
data/CHANGELOG.md ADDED
@@ -0,0 +1,300 @@
1
+ # Rdkafka Changelog
2
+
3
+ ## 0.25.1 (2026-02-26)
4
+ - [Enhancement] Use native ARM64 runners instead of QEMU emulation for Alpine musl aarch64 builds, improving build performance and reliability.
5
+ - [Enhancement] Enable parallel compilation (`make -j$(nproc)`) for ARM64 Alpine musl builds.
6
+ - [Enhancement] Add file descriptor API for fiber scheduler integration. Expose `enable_queue_io_events` and `enable_background_queue_io_events` on `Consumer`, `Producer`, and `Admin` to enable non-blocking monitoring with select/poll/epoll for integration with Ruby fiber schedulers (Falcon, Async) and custom event loops.
7
+ - [Enhancement] Add non-blocking poll methods (`poll_nb`, `events_poll_nb`) on `Consumer` that skip GVL release for efficient fiber scheduler integration when using `poll(0)`.
8
+ - [Enhancement] Add `events_poll_nb_each` method on `Producer`, `Consumer`, and `Admin` for polling events in a single GVL/mutex session. Yields count after each iteration, caller returns `:stop` to break.
9
+ - [Enhancement] Add `poll_nb_each` method on `Consumer` for non-blocking message polling with proper resource cleanup, yielding each message and supporting early termination via `:stop` return value.
10
+
11
+ ## 0.25.0 (2026-01-20)
12
+ - **[Deprecation]** `AbstractHandle#wait` parameter `max_wait_timeout:` (seconds) is deprecated in favor of `max_wait_timeout_ms:` (milliseconds). The old parameter still works but will be removed in v1.0.0.
13
+ - **[Deprecation]** `PartitionsCountCache` constructor parameter `ttl` (seconds) is deprecated in favor of `ttl_ms:` (milliseconds). The old parameter still works but will be removed in v1.0.0.
14
+ - [Enhancement] Extract all timeout defaults to `Rdkafka::Defaults` module for discoverability and per-call overrides (#310). All time-related values are now in milliseconds for consistency.
15
+ - [Enhancement] Add `timeout_ms` parameter to `Consumer#each` for configurable poll timeout.
16
+ - [Enhancement] Extract non-time configuration values (`METADATA_MAX_RETRIES`, `PARTITIONS_COUNT_CACHE_TTL_MS`) to `Rdkafka::Defaults` module.
17
+ - [Enhancement] Bump librdkafka to `2.12.1`
18
+ - [Enhancement] Add descriptive error messages for glibc compatibility issues with instructions for resolution (#654)
19
+ - [Enhancement] Replace magic numbers with named constants throughout codebase for improved readability and maintainability
20
+
21
+ ## 0.24.2 (2025-10-31)
22
+ - [Enhancement] Force lock FFI to 1.17.1 or higher to include critical bug fixes around GCC, write barriers, and thread restarts for forks.
23
+ - [Fix] Fix for Core dump when providing extensions to oauthbearer_set_token (dssjoblom)
24
+
25
+ ## 0.24.1 (2025-10-10)
26
+ - [Fix] Fix Github Action Ruby reference preventing non-compiled releases.
27
+
28
+ ## 0.24.0 (2025-10-10)
29
+ - [Enhancement] Bump librdkafka to `2.11.1`
30
+
31
+ ## 0.23.1 (2025-09-25)
32
+ - [Enhancement] Improve sigstore attestation for precompiled releases.
33
+ - [Fix] Fix incorrectly set default SSL certs dir.
34
+ - [Fix] Disable OpenSSL Heartbeats during compilation.
35
+
36
+ ## 0.23.0 (2025-09-04)
37
+ - **[EOL]** Drop support for Ruby 3.1 to move forward with the fiber scheduler work.
38
+ - [Enhancement] Bump librdkafka to `2.11.0`
39
+ - [Enhancement] Support explicit Debian testing due to lib issues.
40
+ - [Enhancement] Support ARM64 Gnu precompilation.
41
+ - [Enhancement] Improve what symbols are exposed outside of the precompiled extensions.
42
+ - [Enhancement] Introduce an integration suite layer for non RSpec specs execution.
43
+ - [Fix] Add `json` gem as a dependency (was missing but used).
44
+
45
+ ## 0.22.2 (2025-07-21)
46
+ - [Enhancement] Drastically increase number of platforms in the integration suite
47
+ - [Fix] Support Ubuntu `22.04` and older Alpine precompiled versions
48
+ - [Fix] FFI::DynamicLibrary.load_library': Could not open library
49
+ - [Change] Add new CI action to trigger auto-doc refresh.
50
+
51
+ ## 0.22.1 (2025-07-17)
52
+ - [Fix] Fix `Rakefile` being available in the precompiled versions causing build failures.
53
+
54
+ ## 0.22.0 (2025-07-17)
55
+ - **[Feature]** Add precompiled `x86_64-linux-gnu` setup.
56
+ - **[Feature]** Add precompiled `x86_64-linux-musl` setup.
57
+ - **[Feature]** Add precompiled `macos_arm64` setup.
58
+ - [Fix] Fix a case where using empty key on the `musl` architecture would cause a segfault.
59
+ - [Fix] Fix for null pointer reference bypass on empty string being too wide causing segfault.
60
+ - [Enhancement] Allow for producing to non-existing topics with `key` and `partition_key` present.
61
+ - [Enhancement] Replace TTL-based partition count cache with a global cache that reuses `librdkafka` statistics data when possible.
62
+ - [Enhancement] Support producing and consuming of headers with mulitple values (KIP-82).
63
+ - [Enhancement] Allow native Kafka customization poll time.
64
+ - [Enhancement] Roll out experimental jruby support.
65
+ - [Enhancement] Run all specs on each of the platforms with and without precompilation.
66
+ - [Enhancement] Support transactional id in the ACL API.
67
+ - [Fix] Fix issue where post-closed producer C topics refs would not be cleaned.
68
+ - [Fix] Fiber causes Segmentation Fault.
69
+ - [Change] Move to trusted-publishers and remove signing since no longer needed.
70
+
71
+ **Note**: Precompiled extensions are a new feature in this release. While they significantly improve installation speed and reduce build dependencies, they should be thoroughly tested in your staging environment before deploying to production. If you encounter any issues with precompiled extensions, you can fall back to building from sources. For more information, see the [Native Extensions documentation](https://karafka.io/docs/Development-Native-Extensions/).
72
+
73
+ ## 0.21.0 (2025-02-13)
74
+ - [Enhancement] Bump librdkafka to `2.8.0`
75
+
76
+ ## 0.20.0 (2025-01-07)
77
+ - **[Breaking]** Deprecate and remove `#each_batch` due to data consistency concerns.
78
+ - [Enhancement] Bump librdkafka to `2.6.1`
79
+ - [Enhancement] Expose `rd_kafka_global_init` to mitigate macos forking issues.
80
+ - [Enhancement] Avoid clobbering LDFLAGS and CPPFLAGS if in a nix prepared environment (secobarbital).
81
+ - [Patch] Retire no longer needed cooperative-sticky patch.
82
+
83
+ ## 0.19.0 (2024-10-01)
84
+ - **[EOL]** Drop Ruby 3.0 support
85
+ - [Enhancement] Update `librdkafka` to `2.5.3`
86
+ - [Enhancement] Use default oauth callback if none is passed (bachmanity1)
87
+ - [Fix] Fix incorrectly behaving CI on failures.
88
+ - [Patch] Patch with "Add forward declaration to fix compilation without ssl" fix
89
+
90
+ ## 0.18.0 (2024-09-02)
91
+ - [Enhancement] Update `librdkafka` to `2.5.0`
92
+ - [Enhancement] Do not release GVL on `rd_kafka_name` (ferrous26)
93
+ - [Patch] Patch cooperative-sticky assignments in librdkafka.
94
+ - [Fix] Mitigate a case where FFI would not restart the background events callback dispatcher in forks
95
+ - [Fix] Fix unused variable reference in producer (lucasmvnascimento)
96
+
97
+ ## 0.17.0 (2024-08-03)
98
+ - [Feature] Add `#seek_by` to be able to seek for a message by topic, partition and offset (zinahia)
99
+ - [Enhancement] Update `librdkafka` to `2.4.0`
100
+ - [Enhancement] Support ability to release patches to librdkafka.
101
+ - [Change] Remove old producer timeout API warnings.
102
+ - [Fix] Switch to local release of librdkafka to mitigate its unavailability.
103
+
104
+ ## 0.16.1 (2024-07-10)
105
+ - [Fix] Switch to local release of librdkafka to mitigate its unavailability.
106
+
107
+ ## 0.16.0 (2024-06-13)
108
+ - **[EOL]** Retire support for Ruby 2.7.
109
+ - **[Breaking]** Messages without headers returned by `#poll` contain frozen empty hash.
110
+ - **[Breaking]** `HashWithSymbolKeysTreatedLikeStrings` has been removed so headers are regular hashes with string keys.
111
+ - **[Feature]** Support incremental config describe + alter API.
112
+ - **[Feature]** Oauthbearer token refresh callback (bruce-szalwinski-he)
113
+ - **[Feature]** Provide ability to use topic config on a producer for custom behaviors per dispatch.
114
+ - [Enhancement] Use topic config reference cache for messages production to prevent topic objects allocation with each message.
115
+ - [Enhancement] Provide `Rrdkafka::Admin#describe_errors` to get errors descriptions (mensfeld)
116
+ - [Enhancement] Replace time poll based wait engine with an event based to improve response times on blocking operations and wait (nijikon + mensfeld)
117
+ - [Enhancement] Allow for usage of the second regex engine of librdkafka by setting `RDKAFKA_DISABLE_REGEX_EXT` during build (mensfeld)
118
+ - [Enhancement] name polling Thread as `rdkafka.native_kafka#<name>` (nijikon)
119
+ - [Enhancement] Save two objects on message produced and lower CPU usage on message produced with small improvements.
120
+ - [Change] Allow for native kafka thread operations deferring and manual start for consumer, producer and admin.
121
+ - [Change] The `wait_timeout` argument in `AbstractHandle.wait` method is deprecated and will be removed in future versions without replacement. We don't rely on it's value anymore (nijikon)
122
+ - [Fix] Background logger stops working after forking causing memory leaks (mensfeld)
123
+ - [Fix] Fix bogus case/when syntax. Levels 1, 2, and 6 previously defaulted to UNKNOWN (jjowdy)
124
+
125
+ ## 0.15.2 (2024-07-10)
126
+ - [Fix] Switch to local release of librdkafka to mitigate its unavailability.
127
+
128
+ ## 0.15.1 (2024-01-30)
129
+ - [Enhancement] Provide support for Nix OS (alexandriainfantino)
130
+ - [Enhancement] Replace `rd_kafka_offset_store` with `rd_kafka_offsets_store` (mensfeld)
131
+ - [Enhancement] Alias `topic_name` as `topic` in the delivery report (mensfeld)
132
+ - [Enhancement] Provide `label` producer handler and report reference for improved traceability (mensfeld)
133
+ - [Enhancement] Include the error when invoking `create_result` on producer handle (mensfeld)
134
+ - [Enhancement] Skip intermediate array creation on delivery report callback execution (one per message) (mensfeld).
135
+ - [Enhancement] Report `-1` instead of `nil` in case `partition_count` failure (mensfeld).
136
+ - [Fix] Fix return type on `#rd_kafka_poll` (mensfeld)
137
+ - [Fix] `uint8_t` does not exist on Apple Silicon (mensfeld)
138
+ - [Fix] Missing ACL `RD_KAFKA_RESOURCE_BROKER` constant reference (mensfeld)
139
+ - [Fix] Partition cache caches invalid nil result for `PARTITIONS_COUNT_TTL` (mensfeld)
140
+ - [Change] Rename `matching_acl_pattern_type` to `matching_acl_resource_pattern_type` to align the whole API (mensfeld)
141
+
142
+ ## 0.15.0 (2023-12-03)
143
+ - **[Feature]** Add `Admin#metadata` (mensfeld)
144
+ - **[Feature]** Add `Admin#create_partitions` (mensfeld)
145
+ - **[Feature]** Add `Admin#delete_group` utility (piotaixr)
146
+ - **[Feature]** Add Create and Delete ACL Feature To Admin Functions (vgnanasekaran)
147
+ - **[Feature]** Support `#assignment_lost?` on a consumer to check for involuntary assignment revocation (mensfeld)
148
+ - [Enhancement] Expose alternative way of managing consumer events via a separate queue (mensfeld)
149
+ - [Enhancement] **Bump** librdkafka to 2.3.0 (mensfeld)
150
+ - [Enhancement] Increase the `#lag` and `#query_watermark_offsets` default timeouts from 100ms to 1000ms. This will compensate for network glitches and remote clusters operations (mensfeld)
151
+ - [Change] Use `SecureRandom.uuid` instead of `random` for test consumer groups (mensfeld)
152
+
153
+ ## 0.14.1 (2024-07-10)
154
+ - [Fix] Switch to local release of librdkafka to mitigate its unavailability.
155
+
156
+ ## 0.14.0 (2023-11-21)
157
+ - [Enhancement] Add `raise_response_error` flag to the `Rdkafka::AbstractHandle`.
158
+ - [Enhancement] Allow for setting `statistics_callback` as nil to reset predefined settings configured by a different gem (mensfeld)
159
+ - [Enhancement] Get consumer position (thijsc & mensfeld)
160
+ - [Enhancement] Provide `#purge` to remove any outstanding requests from the producer (mensfeld)
161
+ - [Enhancement] Update `librdkafka` to `2.2.0` (mensfeld)
162
+ - [Enhancement] Introduce producer partitions count metadata cache (mensfeld)
163
+ - [Enhancement] Increase metadata timeout request from `250 ms` to `2000 ms` default to allow for remote cluster operations via `rdkafka-ruby` (mensfeld)
164
+ - [Enhancement] Introduce `#name` for producers and consumers (mensfeld)
165
+ - [Enhancement] Include backtrace in non-raised binded errors (mensfeld)
166
+ - [Fix] Reference to Opaque is not released when Admin, Consumer or Producer is closed (mensfeld)
167
+ - [Fix] Trigger `#poll` on native kafka creation to handle oauthbearer cb (mensfeld)
168
+ - [Fix] `#flush` does not handle the timeouts errors by making it return `true` if all flushed or `false` if failed. We do **not** raise an exception here to keep it backwards compatible (mensfeld)
169
+ - **[EOL]** Remove support for Ruby 2.6 due to it being EOL and WeakMap incompatibilities (mensfeld)
170
+ - [Change] Update Kafka Docker with Confluent KRaft (mensfeld)
171
+ - [Change] Update librdkafka repo reference from edenhill to confluentinc (mensfeld)
172
+
173
+ ## 0.13.1 (2024-07-10)
174
+ - [Fix] Switch to local release of librdkafka to mitigate its unavailability.
175
+
176
+ ## 0.13.0 (2023-07-24)
177
+ - Support cooperative sticky partition assignment in the rebalance callback (methodmissing)
178
+ - Support both string and symbol header keys (ColinDKelley)
179
+ - Handle tombstone messages properly (kgalieva)
180
+ - Add topic name to delivery report (maeve)
181
+ - Allow string partitioner config (mollyegibson)
182
+ - Fix documented type for DeliveryReport#error (jimmydo)
183
+ - Bump librdkafka to 2.0.2 (lmaia)
184
+ - Use finalizers to cleanly exit producer and admin (thijsc)
185
+ - Lock access to the native kafka client (thijsc)
186
+ - Fix potential race condition in multi-threaded producer (mensfeld)
187
+ - Fix leaking FFI resources in specs (mensfeld)
188
+ - Improve specs stability (mensfeld)
189
+ - Make metadata request timeout configurable (mensfeld)
190
+ - call_on_partitions_assigned and call_on_partitions_revoked only get a tpl passed in (thijsc)
191
+
192
+ ## 0.12.1 (2024-07-11)
193
+ - [Fix] Switch to local release of librdkafka to mitigate its unavailability.
194
+
195
+ ## 0.12.0 (2022-06-17)
196
+ - Bumps librdkafka to 1.9.0
197
+ - Fix crash on empty partition key (mensfeld)
198
+ - Pass the delivery handle to the callback (gvisokinskas)
199
+
200
+ ## 0.11.0 (2021-11-17)
201
+ - Upgrade librdkafka to 1.8.2
202
+ - Bump supported minimum Ruby version to 2.6
203
+ - Better homebrew path detection
204
+
205
+ ## 0.10.0 (2021-09-07)
206
+ - Upgrade librdkafka to 1.5.0
207
+ - Add error callback config
208
+
209
+ ## 0.9.0 (2021-06-23)
210
+ - Fixes for Ruby 3.0
211
+ - Allow any callable object for callbacks (gremerritt)
212
+ - Reduce memory allocations in Rdkafka::Producer#produce (jturkel)
213
+ - Use queue as log callback to avoid unsafe calls from trap context (breunigs)
214
+ - Allow passing in topic configuration on create_topic (dezka)
215
+ - Add each_batch method to consumer (mgrosso)
216
+
217
+ ## 0.8.1 (2020-12-07)
218
+ - Fix topic_flag behaviour and add tests for Metadata (geoff2k)
219
+ - Add topic admin interface (geoff2k)
220
+ - Raise an exception if @native_kafka is nil (geoff2k)
221
+ - Option to use zstd compression (jasonmartens)
222
+
223
+ ## 0.8.0 (2020-06-02)
224
+ - Upgrade librdkafka to 1.4.0
225
+ - Integrate librdkafka metadata API and add partition_key (by Adithya-copart)
226
+ - Ruby 2.7 compatibility fix (by Geoff Thé)A
227
+ - Add error to delivery report (by Alex Stanovsky)
228
+ - Don't override CPPFLAGS and LDFLAGS if already set on Mac (by Hiroshi Hatake)
229
+ - Allow use of Rake 13.x and up (by Tomasz Pajor)
230
+
231
+ ## 0.7.0 (2019-09-21)
232
+ - Bump librdkafka to 1.2.0 (by rob-as)
233
+ - Allow customizing the wait time for delivery report availability (by mensfeld)
234
+
235
+ ## 0.6.0 (2019-07-23)
236
+ - Bump librdkafka to 1.1.0 (by Chris Gaffney)
237
+ - Implement seek (by breunigs)
238
+
239
+ ## 0.5.0 (2019-04-11)
240
+ - Bump librdkafka to 1.0.0 (by breunigs)
241
+ - Add cluster and member information (by dmexe)
242
+ - Support message headers for consumer & producer (by dmexe)
243
+ - Add consumer rebalance listener (by dmexe)
244
+ - Implement pause/resume partitions (by dmexe)
245
+
246
+ ## 0.4.2 (2019-01-12)
247
+ - Delivery callback for producer
248
+ - Document list param of commit method
249
+ - Use default Homebrew openssl location if present
250
+ - Consumer lag handles empty topics
251
+ - End iteration in consumer when it is closed
252
+ - Add support for storing message offsets
253
+ - Add missing runtime dependency to rake
254
+
255
+ ## 0.4.1 (2018-10-19)
256
+ - Bump librdkafka to 0.11.6
257
+
258
+ ## 0.4.0 (2018-09-24)
259
+ - Improvements in librdkafka archive download
260
+ - Add global statistics callback
261
+ - Use Time for timestamps, potentially breaking change if you
262
+ rely on the previous behavior where it returns an integer with
263
+ the number of milliseconds.
264
+ - Bump librdkafka to 0.11.5
265
+ - Implement TopicPartitionList in Ruby so we don't have to keep
266
+ track of native objects.
267
+ - Support committing a topic partition list
268
+ - Add consumer assignment method
269
+
270
+ ## 0.3.5 (2018-01-17)
271
+ - Fix crash when not waiting for delivery handles
272
+ - Run specs on Ruby 2.5
273
+
274
+ ## 0.3.4 (2017-12-05)
275
+ - Bump librdkafka to 0.11.3
276
+
277
+ ## 0.3.3 (2017-10-27)
278
+ - Fix bug that prevent display of `RdkafkaError` message
279
+
280
+ ## 0.3.2 (2017-10-25)
281
+ - `add_topic` now supports using a partition count
282
+ - Add way to make errors clearer with an extra message
283
+ - Show topics in subscribe error message
284
+ - Show partition and topic in query watermark offsets error message
285
+
286
+ ## 0.3.1 (2017-10-23)
287
+ - Bump librdkafka to 0.11.1
288
+ - Officially support ranges in `add_topic` for topic partition list.
289
+ - Add consumer lag calculator
290
+
291
+ ## 0.3.0 (2017-10-17)
292
+ - Move both add topic methods to one `add_topic` in `TopicPartitionList`
293
+ - Add committed offsets to consumer
294
+ - Add query watermark offset to consumer
295
+
296
+ ## 0.2.0 (2017-10-13)
297
+ - Some refactoring and add inline documentation
298
+
299
+ ## 0.1.x (2017-09-10)
300
+ - Initial working version including producing and consuming
data/Gemfile ADDED
@@ -0,0 +1,13 @@
1
+ # frozen_string_literal: true
2
+
3
+ source "https://rubygems.org"
4
+
5
+ gemspec
6
+
7
+ group :development do
8
+ gem "ostruct"
9
+ gem "pry"
10
+ gem "rspec"
11
+ gem "simplecov"
12
+ gem "warning"
13
+ end
data/Gemfile.lint ADDED
@@ -0,0 +1,14 @@
1
+ # frozen_string_literal: true
2
+
3
+ source "https://rubygems.org"
4
+
5
+ # Documentation linting
6
+ gem "yard-lint"
7
+
8
+ # Code style (StandardRB via RuboCop)
9
+ gem "standard"
10
+ gem "standard-performance"
11
+ gem "rubocop-performance"
12
+ gem "rubocop-rspec"
13
+ gem "standard-rspec"
14
+ gem "rubocop-thread_safety"
data/Gemfile.lint.lock ADDED
@@ -0,0 +1,123 @@
1
+ GEM
2
+ remote: https://rubygems.org/
3
+ specs:
4
+ ast (2.4.3)
5
+ json (2.18.0)
6
+ language_server-protocol (3.17.0.5)
7
+ lint_roller (1.1.0)
8
+ parallel (1.27.0)
9
+ parser (3.3.10.1)
10
+ ast (~> 2.4.1)
11
+ racc
12
+ prism (1.8.0)
13
+ racc (1.8.1)
14
+ rainbow (3.1.1)
15
+ regexp_parser (2.11.3)
16
+ rubocop (1.82.1)
17
+ json (~> 2.3)
18
+ language_server-protocol (~> 3.17.0.2)
19
+ lint_roller (~> 1.1.0)
20
+ parallel (~> 1.10)
21
+ parser (>= 3.3.0.2)
22
+ rainbow (>= 2.2.2, < 4.0)
23
+ regexp_parser (>= 2.9.3, < 3.0)
24
+ rubocop-ast (>= 1.48.0, < 2.0)
25
+ ruby-progressbar (~> 1.7)
26
+ unicode-display_width (>= 2.4.0, < 4.0)
27
+ rubocop-ast (1.49.0)
28
+ parser (>= 3.3.7.2)
29
+ prism (~> 1.7)
30
+ rubocop-capybara (2.22.1)
31
+ lint_roller (~> 1.1)
32
+ rubocop (~> 1.72, >= 1.72.1)
33
+ rubocop-factory_bot (2.28.0)
34
+ lint_roller (~> 1.1)
35
+ rubocop (~> 1.72, >= 1.72.1)
36
+ rubocop-performance (1.26.1)
37
+ lint_roller (~> 1.1)
38
+ rubocop (>= 1.75.0, < 2.0)
39
+ rubocop-ast (>= 1.47.1, < 2.0)
40
+ rubocop-rspec (3.9.0)
41
+ lint_roller (~> 1.1)
42
+ rubocop (~> 1.81)
43
+ rubocop-rspec_rails (2.32.0)
44
+ lint_roller (~> 1.1)
45
+ rubocop (~> 1.72, >= 1.72.1)
46
+ rubocop-rspec (~> 3.5)
47
+ rubocop-thread_safety (0.7.3)
48
+ lint_roller (~> 1.1)
49
+ rubocop (~> 1.72, >= 1.72.1)
50
+ rubocop-ast (>= 1.44.0, < 2.0)
51
+ ruby-progressbar (1.13.0)
52
+ standard (1.53.0)
53
+ language_server-protocol (~> 3.17.0.2)
54
+ lint_roller (~> 1.0)
55
+ rubocop (~> 1.82.0)
56
+ standard-custom (~> 1.0.0)
57
+ standard-performance (~> 1.8)
58
+ standard-custom (1.0.2)
59
+ lint_roller (~> 1.0)
60
+ rubocop (~> 1.50)
61
+ standard-performance (1.9.0)
62
+ lint_roller (~> 1.1)
63
+ rubocop-performance (~> 1.26.0)
64
+ standard-rspec (0.3.1)
65
+ lint_roller (>= 1.0)
66
+ rubocop-capybara (~> 2.22)
67
+ rubocop-factory_bot (~> 2.27)
68
+ rubocop-rspec (~> 3.5)
69
+ rubocop-rspec_rails (~> 2.31)
70
+ unicode-display_width (3.2.0)
71
+ unicode-emoji (~> 4.1)
72
+ unicode-emoji (4.2.0)
73
+ yard (0.9.38)
74
+ yard-lint (1.4.0)
75
+ yard (~> 0.9)
76
+ zeitwerk (~> 2.6)
77
+ zeitwerk (2.7.4)
78
+
79
+ PLATFORMS
80
+ ruby
81
+ x86_64-linux
82
+
83
+ DEPENDENCIES
84
+ rubocop-performance
85
+ rubocop-rspec
86
+ rubocop-thread_safety
87
+ standard
88
+ standard-performance
89
+ standard-rspec
90
+ yard-lint
91
+
92
+ CHECKSUMS
93
+ ast (2.4.3) sha256=954615157c1d6a382bc27d690d973195e79db7f55e9765ac7c481c60bdb4d383
94
+ json (2.18.0) sha256=b10506aee4183f5cf49e0efc48073d7b75843ce3782c68dbeb763351c08fd505
95
+ language_server-protocol (3.17.0.5) sha256=fd1e39a51a28bf3eec959379985a72e296e9f9acfce46f6a79d31ca8760803cc
96
+ lint_roller (1.1.0) sha256=2c0c845b632a7d172cb849cc90c1bce937a28c5c8ccccb50dfd46a485003cc87
97
+ parallel (1.27.0) sha256=4ac151e1806b755fb4e2dc2332cbf0e54f2e24ba821ff2d3dcf86bf6dc4ae130
98
+ parser (3.3.10.1) sha256=06f6a725d2cd91e5e7f2b7c32ba143631e1f7c8ae2fb918fc4cebec187e6a688
99
+ prism (1.8.0) sha256=84453a16ef5530ea62c5f03ec16b52a459575ad4e7b9c2b360fd8ce2c39c1254
100
+ racc (1.8.1) sha256=4a7f6929691dbec8b5209a0b373bc2614882b55fc5d2e447a21aaa691303d62f
101
+ rainbow (3.1.1) sha256=039491aa3a89f42efa1d6dec2fc4e62ede96eb6acd95e52f1ad581182b79bc6a
102
+ regexp_parser (2.11.3) sha256=ca13f381a173b7a93450e53459075c9b76a10433caadcb2f1180f2c741fc55a4
103
+ rubocop (1.82.1) sha256=09f1a6a654a960eda767aebea33e47603080f8e9c9a3f019bf9b94c9cab5e273
104
+ rubocop-ast (1.49.0) sha256=49c3676d3123a0923d333e20c6c2dbaaae2d2287b475273fddee0c61da9f71fd
105
+ rubocop-capybara (2.22.1) sha256=ced88caef23efea53f46e098ff352f8fc1068c649606ca75cb74650970f51c0c
106
+ rubocop-factory_bot (2.28.0) sha256=4b17fc02124444173317e131759d195b0d762844a71a29fe8139c1105d92f0cb
107
+ rubocop-performance (1.26.1) sha256=cd19b936ff196df85829d264b522fd4f98b6c89ad271fa52744a8c11b8f71834
108
+ rubocop-rspec (3.9.0) sha256=8fa70a3619408237d789aeecfb9beef40576acc855173e60939d63332fdb55e2
109
+ rubocop-rspec_rails (2.32.0) sha256=4a0d641c72f6ebb957534f539d9d0a62c47abd8ce0d0aeee1ef4701e892a9100
110
+ rubocop-thread_safety (0.7.3) sha256=067cdd52fbf5deffc18995437e45b5194236eaff4f71de3375a1f6052e48f431
111
+ ruby-progressbar (1.13.0) sha256=80fc9c47a9b640d6834e0dc7b3c94c9df37f08cb072b7761e4a71e22cff29b33
112
+ standard (1.53.0) sha256=f3c9493385db7079d0abce6f7582f553122156997b81258cd361d3480eeacf9c
113
+ standard-custom (1.0.2) sha256=424adc84179a074f1a2a309bb9cf7cd6bfdb2b6541f20c6bf9436c0ba22a652b
114
+ standard-performance (1.9.0) sha256=49483d31be448292951d80e5e67cdcb576c2502103c7b40aec6f1b6e9c88e3f2
115
+ standard-rspec (0.3.1) sha256=67bc957281cacf24f0d88235ca1bf28a8995265b1a60eb519cd0451858b56a22
116
+ unicode-display_width (3.2.0) sha256=0cdd96b5681a5949cdbc2c55e7b420facae74c4aaf9a9815eee1087cb1853c42
117
+ unicode-emoji (4.2.0) sha256=519e69150f75652e40bf736106cfbc8f0f73aa3fb6a65afe62fefa7f80b0f80f
118
+ yard (0.9.38) sha256=721fb82afb10532aa49860655f6cc2eaa7130889df291b052e1e6b268283010f
119
+ yard-lint (1.4.0) sha256=7dd88fbb08fd77cb840bea899d58812817b36d92291b5693dd0eeb3af9f91f0f
120
+ zeitwerk (2.7.4) sha256=2bef90f356bdafe9a6c2bd32bcd804f83a4f9b8bc27f3600fff051eb3edcec8b
121
+
122
+ BUNDLED WITH
123
+ 4.0.3
data/MIT-LICENSE ADDED
@@ -0,0 +1,22 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2017-2023 Thijs Cadier
4
+ 2023, Maciej Mensfeld
5
+
6
+ Permission is hereby granted, free of charge, to any person obtaining a copy
7
+ of this software and associated documentation files (the "Software"), to deal
8
+ in the Software without restriction, including without limitation the rights
9
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
10
+ copies of the Software, and to permit persons to whom the Software is
11
+ furnished to do so, subject to the following conditions:
12
+
13
+ The above copyright notice and this permission notice shall be included in all
14
+ copies or substantial portions of the Software.
15
+
16
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
17
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
18
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
19
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
20
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
21
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
22
+ SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,181 @@
1
+ # Rdkafka
2
+
3
+ [![Build Status](https://github.com/karafka/rdkafka-ruby/actions/workflows/ci_linux_ubuntu_x86_64_gnu.yml/badge.svg)](https://github.com/karafka/rdkafka-ruby/actions/workflows/ci_linux_x86_64_gnu.yml)
4
+ [![Gem Version](https://badge.fury.io/rb/rdkafka.svg)](https://badge.fury.io/rb/rdkafka)
5
+ [![Join the chat at https://slack.karafka.io](https://raw.githubusercontent.com/karafka/misc/master/slack.svg)](https://slack.karafka.io)
6
+
7
+ > [!NOTE]
8
+ > The `rdkafka-ruby` gem was created and developed by [AppSignal](https://www.appsignal.com/). Their impactful contributions have significantly shaped the Ruby Kafka and Karafka ecosystems. For robust monitoring, we highly recommend AppSignal.
9
+
10
+ ---
11
+
12
+ The `rdkafka` gem is a modern Kafka client library for Ruby based on
13
+ [librdkafka](https://github.com/confluentinc/librdkafka/).
14
+ It wraps the production-ready C client using the [ffi](https://github.com/ffi/ffi)
15
+ gem and targets Kafka 1.0+ and Ruby versions under security or
16
+ active maintenance. We remove a Ruby version from our CI builds when they
17
+ become EOL.
18
+
19
+ `rdkafka` was written because of the need for a reliable Ruby client for Kafka that supports modern Kafka at [AppSignal](https://appsignal.com). AppSignal runs it in production on very high-traffic systems.
20
+
21
+ The most essential pieces of a Kafka client are implemented, and we aim to provide all relevant consumer, producer, and admin APIs.
22
+
23
+ ## Table of content
24
+
25
+ - [Project Scope](#project-scope)
26
+ - [Installation](#installation)
27
+ - [Usage](#usage)
28
+ * [Consuming Messages](#consuming-messages)
29
+ * [Producing Messages](#producing-messages)
30
+ - [Higher Level Libraries](#higher-level-libraries)
31
+ * [Message Processing Frameworks](#message-processing-frameworks)
32
+ * [Message Publishing Libraries](#message-publishing-libraries)
33
+ - [Forking](#forking)
34
+ - [Development](#development)
35
+ - [Example](#example)
36
+ - [Versions](#versions)
37
+
38
+ ## Project Scope
39
+
40
+ While rdkafka-ruby aims to simplify the use of librdkafka in Ruby applications, it's important to understand the limitations of this library:
41
+
42
+ - **No Complex Producers/Consumers**: This library does not intend to offer complex producers or consumers. The aim is to stick closely to the functionalities provided by librdkafka itself.
43
+
44
+ - **Focus on librdkafka Capabilities**: Features that can be achieved directly in Ruby, without specific needs from librdkafka, are outside the scope of this library.
45
+
46
+ - **Existing High-Level Functionalities**: Certain high-level functionalities like producer metadata cache and simple consumer are already part of the library. Although they fall slightly outside the primary goal, they will remain part of the contract, given their existing usage.
47
+
48
+
49
+ ## Installation
50
+
51
+ When installed, this gem downloads and compiles librdkafka. If you have any problems installing the gem, please open an issue.
52
+
53
+ ## Usage
54
+
55
+ Please see the [documentation](https://karafka.io/docs/code/rdkafka-ruby/) for full details on how to use this gem. Below are two quick examples.
56
+
57
+ Unless you are seeking specific low-level capabilities, we **strongly** recommend using [Karafka](https://github.com/karafka/karafka) and [WaterDrop](https://github.com/karafka/waterdrop) when working with Kafka. These are higher-level libraries also maintained by us based on rdkafka-ruby.
58
+
59
+ ### Consuming Messages
60
+
61
+ Subscribe to a topic and get messages. Kafka will automatically spread
62
+ the available partitions over consumers with the same group id.
63
+
64
+ ```ruby
65
+ config = {
66
+ :"bootstrap.servers" => "localhost:9092",
67
+ :"group.id" => "ruby-test"
68
+ }
69
+ consumer = Rdkafka::Config.new(config).consumer
70
+ consumer.subscribe("ruby-test-topic")
71
+
72
+ consumer.each do |message|
73
+ puts "Message received: #{message}"
74
+ end
75
+ ```
76
+
77
+ ### Producing Messages
78
+
79
+ Produce several messages, put the delivery handles in an array, and
80
+ wait for them before exiting. This way the messages will be batched and
81
+ efficiently sent to Kafka.
82
+
83
+ ```ruby
84
+ config = {:"bootstrap.servers" => "localhost:9092"}
85
+ producer = Rdkafka::Config.new(config).producer
86
+ delivery_handles = []
87
+
88
+ 100.times do |i|
89
+ puts "Producing message #{i}"
90
+ delivery_handles << producer.produce(
91
+ topic: "ruby-test-topic",
92
+ payload: "Payload #{i}",
93
+ key: "Key #{i}"
94
+ )
95
+ end
96
+
97
+ delivery_handles.each(&:wait)
98
+ ```
99
+
100
+ Note that creating a producer consumes some resources that will not be released until it `#close` is explicitly called, so be sure to call `Config#producer` only as necessary.
101
+
102
+ ## Higher Level Libraries
103
+
104
+ Currently, there are two actively developed frameworks based on `rdkafka-ruby`, that provide higher-level API that can be used to work with Kafka messages and one library for publishing messages.
105
+
106
+ ### Message Processing Frameworks
107
+
108
+ * [Karafka](https://github.com/karafka/karafka) - Ruby and Rails efficient Kafka processing framework.
109
+ * [Racecar](https://github.com/zendesk/racecar) - A simple framework for Kafka consumers in Ruby
110
+
111
+ ### Message Publishing Libraries
112
+
113
+ * [WaterDrop](https://github.com/karafka/waterdrop) – Standalone Karafka library for producing Kafka messages.
114
+
115
+ ## Forking
116
+
117
+ When working with `rdkafka-ruby`, it's essential to know that the underlying `librdkafka` library does not support fork-safe operations, even though it is thread-safe. Forking a process after initializing librdkafka clients can lead to unpredictable behavior due to inherited file descriptors and memory states. This limitation requires careful handling, especially in Ruby applications that rely on forking.
118
+
119
+ To address this, it's highly recommended to:
120
+
121
+ - Never initialize any `rdkafka-ruby` producers or consumers before forking to avoid state corruption.
122
+ - Before forking, always close any open producers or consumers if you've opened any.
123
+ - Use high-level libraries like [WaterDrop](https://github.com/karafka/waterdrop) and [Karafka](https://github.com/karafka/karafka/), which provide abstractions for handling librdkafka's intricacies.
124
+
125
+ ## Development
126
+
127
+ Contributors are encouraged to focus on enhancements that align with the core goal of the library. We appreciate contributions but will likely not accept pull requests for features that:
128
+
129
+ - Implement functionalities that can achieved using standard Ruby capabilities without changes to the underlying rdkafka-ruby bindings.
130
+ - Deviate significantly from the primary aim of providing librdkafka bindings with Ruby-friendly interfaces.
131
+
132
+ A Docker Compose file is included to run Kafka. To run that:
133
+
134
+ ```
135
+ docker-compose up
136
+ ```
137
+
138
+ Run `bundle` and `cd ext && bundle exec rake && cd ..` to download and compile `librdkafka`.
139
+
140
+ You can then run `bundle exec rspec` to run the tests. To see rdkafka debug output:
141
+
142
+ ```
143
+ DEBUG_PRODUCER=true bundle exec rspec
144
+ DEBUG_CONSUMER=true bundle exec rspec
145
+ ```
146
+
147
+ After running the tests, you can bring the cluster down to start with a clean slate:
148
+
149
+ ```
150
+ docker-compose down
151
+ ```
152
+
153
+ ## Example
154
+
155
+ To see everything working, run these in separate tabs:
156
+
157
+ ```
158
+ bundle exec rake consume_messages
159
+ bundle exec rake produce_messages
160
+ ```
161
+
162
+ ## Versions
163
+
164
+ | rdkafka-ruby | librdkafka | patches |
165
+ |-|-|-|
166
+ | 0.25.x (2026-01-21) | 2.12.1 (2025-10-21) | yes |
167
+ | 0.24.x (2025-10-10) | 2.11.1 (2025-08-18) | yes |
168
+ | 0.23.x (2025-09-04) | 2.11.0 (2025-07-03) | yes |
169
+ | 0.22.x (2025-07-17) | 2.8.0 (2025-01-07) | yes |
170
+ | 0.21.x (2025-02-13) | 2.8.0 (2025-01-07) | yes |
171
+ | 0.20.0 (2025-01-07) | 2.6.1 (2024-11-18) | yes |
172
+ | 0.19.0 (2024-10-01) | 2.5.3 (2024-09-02) | yes |
173
+ | 0.18.0 (2024-09-02) | 2.5.0 (2024-06-10) | yes |
174
+ | 0.17.0 (2024-08-03) | 2.4.0 (2024-05-07) | no |
175
+ | 0.16.0 (2024-06-13) | 2.3.0 (2023-10-25) | no |
176
+ | 0.15.0 (2023-12-03) | 2.3.0 (2023-10-25) | no |
177
+ | 0.14.0 (2023-11-21) | 2.2.0 (2023-07-12) | no |
178
+ | 0.13.0 (2023-07-24) | 2.0.2 (2023-01-20) | no |
179
+ | 0.12.0 (2022-06-17) | 1.9.0 (2022-06-16) | no |
180
+ | 0.11.0 (2021-11-17) | 1.8.2 (2021-10-18) | no |
181
+ | 0.10.0 (2021-09-07) | 1.5.0 (2020-07-20) | no |