karafka-rdkafka 0.16.1 → 0.17.1.beta1

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: f8685868c1ad3f222639845933ba5aec660ec3923465c4cf91c3235ece03eec8
4
- data.tar.gz: 96d7c33291d166418e696f05cce117b89861dd7fbbf53b587dbf1a90a3f4d2b2
3
+ metadata.gz: 192d225abdb3819438f2da32dde41f7940666ee105f51f315ab55cf670e8c3b9
4
+ data.tar.gz: bc393b0baa8f62980b9acf4d8d993f63cc1f07f9e3c8da54bceb8a76b387849b
5
5
  SHA512:
6
- metadata.gz: 30de272e1cac622957ecc061426bf7314791528769dc6afc3f445f5fe6b3241a5b51a88931c97619fcdf1447bd0d6b7913c959ccb3b5d384e32d5ce89ef5d915
7
- data.tar.gz: f912ec91dd3983a89807879b272f8bbcf777f705bd305aba77148998afcf2ba4372ae070d2510a19110a424ad9231177c64a25742b5de47b22c426769da5c858
6
+ metadata.gz: 82476a66a8ae78150ee80122df8d31925afb31ba06bc9cd0467ebf2fa09fcacaaa69f0262e9ff75447461871c29c3db9026cc3adda505cd9b0fe9dd8741f3336
7
+ data.tar.gz: 86fca73dcffd68a5197276a01cf6366ef4f0ba702656e78209e8666534f76a1bb3f05f5e624f812ed6576255545093b810603be267f2a7653e7005ea42c1ddc0
checksums.yaml.gz.sig CHANGED
Binary file
data/.ruby-version CHANGED
@@ -1 +1 @@
1
- 3.3.3
1
+ 3.3.4
data/CHANGELOG.md CHANGED
@@ -1,5 +1,12 @@
1
1
  # Rdkafka Changelog
2
2
 
3
+ ## 0.17.1 (Unreleased)
4
+ - [Enhancement] Support ability to release patches to librdkafka.
5
+ - [Patch] Patch cooperative-sticky assignments in librdkafka.
6
+
7
+ ## 0.17.0 (2024-07-21)
8
+ - [Enhancement] Bump librdkafka to 2.5.0
9
+
3
10
  ## 0.16.1 (2024-07-10)
4
11
  - [Feature] Add `#seek_by` to be able to seek for a message by topic, partition and offset (zinahia)
5
12
  - [Fix] Switch to local release of librdkafka to mitigate its unavailability.
@@ -11,6 +18,9 @@
11
18
  - [Enhancement] Save two objects on message produced and lower CPU usage on message produced with small improvements.
12
19
  - [Fix] Remove support for Ruby 2.7. Supporting it was a bug since rest of the karafka ecosystem no longer supports it.
13
20
 
21
+ ## 0.15.2 (2024-07-10)
22
+ - [Fix] Switch to local release of librdkafka to mitigate its unavailability.
23
+
14
24
  ## 0.15.1 (2024-05-09)
15
25
  - **[Feature]** Provide ability to use topic config on a producer for custom behaviors per dispatch.
16
26
  - [Enhancement] Use topic config reference cache for messages production to prevent topic objects allocation with each message.
@@ -20,6 +30,15 @@
20
30
  - **[Feature]** Oauthbearer token refresh callback (bruce-szalwinski-he)
21
31
  - **[Feature]** Support incremental config describe + alter API (mensfeld)
22
32
  - [Enhancement] name polling Thread as `rdkafka.native_kafka#<name>` (nijikon)
33
+ - [Enhancement] Replace time poll based wait engine with an event based to improve response times on blocking operations and wait (nijikon + mensfeld)
34
+ - [Enhancement] Allow for usage of the second regex engine of librdkafka by setting `RDKAFKA_DISABLE_REGEX_EXT` during build (mensfeld)
35
+ - [Enhancement] name polling Thread as `rdkafka.native_kafka#<name>` (nijikon)
36
+ - [Change] Allow for native kafka thread operations deferring and manual start for consumer, producer and admin.
37
+ - [Change] The `wait_timeout` argument in `AbstractHandle.wait` method is deprecated and will be removed in future versions without replacement. We don't rely on it's value anymore (nijikon)
38
+ - [Fix] Fix bogus case/when syntax. Levels 1, 2, and 6 previously defaulted to UNKNOWN (jjowdy)
39
+
40
+ ## 0.14.11 (2024-07-10)
41
+ - [Fix] Switch to local release of librdkafka to mitigate its unavailability.
23
42
 
24
43
  ## 0.14.10 (2024-02-08)
25
44
  - [Fix] Background logger stops working after forking causing memory leaks (mensfeld).
@@ -67,6 +86,9 @@
67
86
  - [Enhancement] Bump librdkafka to 2.3.0
68
87
  - [Enhancement] Increase the `#lag` and `#query_watermark_offsets` default timeouts from 100ms to 1000ms. This will compensate for network glitches and remote clusters operations.
69
88
 
89
+ ## 0.13.10 (2024-07-10)
90
+ - [Fix] Switch to local release of librdkafka to mitigate its unavailability.
91
+
70
92
  ## 0.13.9 (2023-11-07)
71
93
  - [Enhancement] Expose alternative way of managing consumer events via a separate queue.
72
94
  - [Enhancement] Allow for setting `statistics_callback` as nil to reset predefined settings configured by a different gem.
@@ -121,6 +143,9 @@
121
143
  - Retry metadta fetches on certain errors with a backoff (mensfeld)
122
144
  - Do not lock access to underlying native kafka client and rely on Karafka granular locking (mensfeld)
123
145
 
146
+ ## 0.12.4 (2024-07-10)
147
+ - [Fix] Switch to local release of librdkafka to mitigate its unavailability.
148
+
124
149
  ## 0.12.3
125
150
  - Include backtrace in non-raised binded errors.
126
151
  - Include topic name in the delivery reports
data/README.md CHANGED
@@ -163,8 +163,8 @@ bundle exec rake produce_messages
163
163
 
164
164
  | rdkafka-ruby | librdkafka |
165
165
  |-|-|
166
- | 0.17.0 (Unreleased) | 2.4.0 (2024-05-07) |
167
- | 0.16.0 (2024-06-13) | 2.3.0 (2023-10-25) |
166
+ | 0.17.0 (Unreleased) | 2.5.0 (2024-07-10) |
167
+ | 0.16.0 (2024-06-13) | 2.4.0 (2024-05-07) |
168
168
  | 0.15.0 (2023-12-03) | 2.3.0 (2023-10-25) |
169
169
  | 0.14.0 (2023-11-21) | 2.2.0 (2023-07-12) |
170
170
  | 0.13.0 (2023-07-24) | 2.0.2 (2023-01-20) |
@@ -0,0 +1,26 @@
1
+ # This patch is released under the 2-clause BSD license, same as librdkafka
2
+ # Fixes: https://github.com/confluentinc/librdkafka/issues/4783
3
+ #
4
+ --- librdkafka_2.5.0/src/rdkafka_sticky_assignor.c 2024-07-08 09:47:43.000000000 +0200
5
+ +++ librdkafka_2.5.0/src/rdkafka_sticky_assignor.c 2024-07-30 09:44:38.529759640 +0200
6
+ @@ -769,7 +769,7 @@
7
+ const rd_kafka_topic_partition_list_t *partitions;
8
+ const char *consumer;
9
+ const rd_map_elem_t *elem;
10
+ - int i;
11
+ + int i, j;
12
+
13
+ /* The assignment is balanced if minimum and maximum numbers of
14
+ * partitions assigned to consumers differ by at most one. */
15
+ @@ -836,9 +836,9 @@
16
+
17
+ /* Otherwise make sure it can't get any more partitions */
18
+
19
+ - for (i = 0; i < potentialTopicPartitions->cnt; i++) {
20
+ + for (j = 0; j < potentialTopicPartitions->cnt; j++) {
21
+ const rd_kafka_topic_partition_t *partition =
22
+ - &potentialTopicPartitions->elems[i];
23
+ + &potentialTopicPartitions->elems[j];
24
+ const char *otherConsumer;
25
+ int otherConsumerPartitionCount;
26
+
data/docker-compose.yml CHANGED
@@ -3,7 +3,7 @@ version: '2'
3
3
  services:
4
4
  kafka:
5
5
  container_name: kafka
6
- image: confluentinc/cp-kafka:7.6.1
6
+ image: confluentinc/cp-kafka:7.7.0
7
7
 
8
8
  ports:
9
9
  - 9092:9092
data/ext/README.md CHANGED
@@ -1,7 +1,7 @@
1
1
  # Ext
2
2
 
3
- This gem depends on the `librdkafka` C library. It is downloaded when
4
- this gem is installed.
3
+ This gem depends on the `librdkafka` C library. It is downloaded, stored in
4
+ `dist/` directory, and checked into source control.
5
5
 
6
6
  To update the `librdkafka` version follow the following steps:
7
7
 
@@ -9,8 +9,9 @@ To update the `librdkafka` version follow the following steps:
9
9
  version number and asset checksum for `tar.gz`.
10
10
  * Change the version in `lib/rdkafka/version.rb`
11
11
  * Change the `sha256` in `lib/rdkafka/version.rb`
12
- * Run `bundle exec rake` in the `ext` directory to download and build
13
- the new version
12
+ * Run `bundle exec rake dist:download` in the `ext` directory to download the
13
+ new release and place it in the `dist/` for you
14
+ * Run `bundle exec rake` in the `ext` directory to build the new version
14
15
  * Run `docker-compose pull` in the main gem directory to ensure the docker
15
16
  images used by the tests and run `docker-compose up`
16
17
  * Finally, run `bundle exec rspec` in the main gem directory to execute
data/ext/Rakefile CHANGED
@@ -1,6 +1,7 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  require File.expand_path('../../lib/rdkafka/version', __FILE__)
4
+ require "digest"
4
5
  require "fileutils"
5
6
  require "open-uri"
6
7
 
@@ -30,6 +31,8 @@ task :default => :clean do
30
31
  }
31
32
  recipe.configure_options = ["--host=#{recipe.host}"]
32
33
 
34
+ recipe.patch_files = Dir[File.join(releases, 'patches', "*.patch")].sort
35
+
33
36
  # Disable using libc regex engine in favor of the embedded one
34
37
  # The default regex engine of librdkafka does not always work exactly as most of the users
35
38
  # would expect, hence this flag allows for changing it to the other one
@@ -71,6 +74,42 @@ task :clean do
71
74
  FileUtils.rm_rf File.join(File.dirname(__FILE__), "tmp")
72
75
  end
73
76
 
77
+ namespace :dist do
78
+ task :dir do
79
+ ENV["RDKAFKA_DIST_PATH"] ||= File.expand_path(File.join(File.dirname(__FILE__), '..', 'dist'))
80
+ end
81
+
82
+ task :file => "dist:dir" do
83
+ ENV["RDKAFKA_DIST_FILE"] ||= File.join(ENV["RDKAFKA_DIST_PATH"], "librdkafka_#{Rdkafka::LIBRDKAFKA_VERSION}.tar.gz")
84
+ end
85
+
86
+ task :clean => "dist:file" do
87
+ Dir.glob(File.join("#{ENV['RDKAFKA_DIST_PATH']}", "*")).each do |filename|
88
+ next if filename.include? ENV["RDKAFKA_DIST_FILE"]
89
+
90
+ FileUtils.rm_rf filename
91
+ end
92
+ end
93
+
94
+ task :download => "dist:file" do
95
+ version = Rdkafka::LIBRDKAFKA_VERSION
96
+ librdkafka_download = "https://codeload.github.com/confluentinc/librdkafka/tar.gz/v#{version}"
97
+
98
+ URI.open(librdkafka_download) do |file|
99
+ filename = ENV["RDKAFKA_DIST_FILE"]
100
+ data = file.read
101
+
102
+ if Digest::SHA256.hexdigest(data) != Rdkafka::LIBRDKAFKA_SOURCE_SHA256
103
+ raise "SHA256 does not match downloaded file"
104
+ end
105
+
106
+ File.write(filename, data)
107
+ end
108
+ end
109
+
110
+ task :update => %w[dist:download dist:clean]
111
+ end
112
+
74
113
  namespace :build do
75
114
  desc "Build librdkafka at the given git sha or tag"
76
115
  task :git, [:ref] do |task, args|
@@ -78,8 +117,9 @@ namespace :build do
78
117
  version = "git-#{ref}"
79
118
 
80
119
  recipe = MiniPortile.new("librdkafka", version)
81
- recipe.files << "https://github.com/edenhill/librdkafka/archive/#{ref}.tar.gz"
120
+ recipe.files << "https://github.com/confluentinc/librdkafka/archive/#{ref}.tar.gz"
82
121
  recipe.configure_options = ["--host=#{recipe.host}","--enable-static", "--enable-zstd"]
122
+ recipe.patch_files = Dir[File.join(releases, 'patches', "*.patch")].sort
83
123
  recipe.cook
84
124
 
85
125
  ext = recipe.host.include?("darwin") ? "dylib" : "so"
@@ -1,7 +1,7 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Rdkafka
4
- VERSION = "0.16.1"
5
- LIBRDKAFKA_VERSION = "2.4.0"
6
- LIBRDKAFKA_SOURCE_SHA256 = "d645e47d961db47f1ead29652606a502bdd2a880c85c1e060e94eea040f1a19a"
4
+ VERSION = "0.17.1.beta1"
5
+ LIBRDKAFKA_VERSION = "2.5.0"
6
+ LIBRDKAFKA_SOURCE_SHA256 = "3dc62de731fd516dfb1032861d9a580d4d0b5b0856beb0f185d06df8e6c26259"
7
7
  end
@@ -34,7 +34,7 @@ describe Rdkafka::Admin do
34
34
  describe '#describe_errors' do
35
35
  let(:errors) { admin.class.describe_errors }
36
36
 
37
- it { expect(errors.size).to eq(168) }
37
+ it { expect(errors.size).to eq(170) }
38
38
  it { expect(errors[-184]).to eq(code: -184, description: 'Local: Queue full', name: '_QUEUE_FULL') }
39
39
  it { expect(errors[21]).to eq(code: 21, description: 'Broker: Invalid required acks value', name: 'INVALID_REQUIRED_ACKS') }
40
40
  end
data.tar.gz.sig CHANGED
Binary file
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: karafka-rdkafka
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.16.1
4
+ version: 0.17.1.beta1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Thijs Cadier
@@ -36,7 +36,7 @@ cert_chain:
36
36
  AnG1dJU+yL2BK7vaVytLTstJME5mepSZ46qqIJXMuWob/YPDmVaBF39TDSG9e34s
37
37
  msG3BiCqgOgHAnL23+CN3Rt8MsuRfEtoTKpJVcCfoEoNHOkc
38
38
  -----END CERTIFICATE-----
39
- date: 2024-07-10 00:00:00.000000000 Z
39
+ date: 2024-07-30 00:00:00.000000000 Z
40
40
  dependencies:
41
41
  - !ruby/object:Gem::Dependency
42
42
  name: ffi
@@ -186,7 +186,8 @@ files:
186
186
  - README.md
187
187
  - Rakefile
188
188
  - certs/cert_chain.pem
189
- - dist/librdkafka_2.4.0.tar.gz
189
+ - dist/librdkafka_2.5.0.tar.gz
190
+ - dist/patches/rdkafka_sticky_assignor.c.patch
190
191
  - docker-compose.yml
191
192
  - ext/README.md
192
193
  - ext/Rakefile
metadata.gz.sig CHANGED
@@ -1 +1,2 @@
1
- *�lp}x�F%���5��|��RUs4zD$c��n�/�r�%�5��鵖%Y�j�O�Շ�9�75�6b��<�tY�[x:{#fj��-�xE�ck�!���Y�̼V�/�����2��X>�[� |��C�h"D�~c)��D9�(y��K>��&��X������G�rtکH�:1�P^�=k[���ыQ�����oaH��'�ŕM��48a�=�ŧ��+(!wEnn��D�G<}�U�U ��],�xmG�U#�m�#���&�-����1Z�)� F#+�$�X���rĆ� J�9<NQ;�UYv������^��VZ�T{�m��.>6 ,�gd �Pؼ��r�s��i�^B�q-�8=l/ �!��k�cq�Sr�
1
+ Y3�&���ikAVt���2�ON׿���� �������H[U�ǡ���R�홥̈;�0�@7x}Վٚ��\Xi�?J")��1��vxI}�궕��5��ڬ��D�M�h]����*³�}��k
2
+ �C�+���DO��9���sO��K��{nk��ڤ