ruby-kafka 0.7.1 → 0.7.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 07f5c2801a1dda787213437b1b27aba5bd23b25e53c64687b5910ee0e1ccc221
-  data.tar.gz: 4f277fc15146a78d2374da0dff2e8201bf521012a07b160fc8f0630164656509
+  metadata.gz: cae08d9f12b1f6758def92d1d02ab046993bb11eed7b985b91be9bd5eee57b51
+  data.tar.gz: 89564d3428e93b5d377d995a8400a27fe07d40e09bb860371c78cca550549d7a
 SHA512:
-  metadata.gz: 58eb8f78f5d2cea9793416ab9e01d74f8c7737e7d7a5174a13a58278fa6b5849a2ee3b88155c97f37915664d0c9f393738ae7b8e06c2b78d48543188514a9cf8
-  data.tar.gz: 714ce12cca6bc8dd34c20835cdbc89257cb0f5a1cb020d1312e66adfc3d0ec91e1cec3207af0d708cdc936afb803d13576b2fe565d73495e45fd7cc89fc90f1f
+  metadata.gz: b602a8d11ceb4a489a091b7c316493a58e3c009d1f02d3b9008e882408bd4ff9c4f5ce51f4f50aec6c9478eab782b617268b206fee76e50ead20eff260bfa6a2
+  data.tar.gz: ddaae218d4d56885b610e89cc9aa9cd70d12ea196e888c5b50a2ff7be7d53e35932403b81c99e131cbaecc9d56340cd607b381d09ed1a25db324440a35e2ddb5

data/CHANGELOG.md

@@ -4,6 +4,13 @@ Changes and additions to the library will be listed here.
 
 ## Unreleased
 
+## 0.7.2
+
+- Handle case when paused partition does not belong to group on resume (#656).
+- Fix compatibility version in documentation (#651).
+- Fix message set backward compatible (#648).
+- Refresh metadata on connection error when listing topics (#644).
+
 ## 0.7.1
 
 - Compatibility with dogstatsd-ruby v4.0.0.

data/README.md

@@ -90,7 +90,7 @@ Or install it yourself as:
   </tr>
   <tr>
     <th>Kafka 0.11</th>
-    <td>Limited support</td>
+    <td>Full support in v0.7.x</td>
     <td>Limited support</td>
   </tr>
   <tr>
@@ -105,8 +105,8 @@ This library is targeting Kafka 0.9 with the v0.4.x series and Kafka 0.10 with t
 - **Kafka 0.8:** Full support for the Producer API in ruby-kafka v0.4.x, but no support for consumer groups. Simple message fetching works.
 - **Kafka 0.9:** Full support for the Producer and Consumer API in ruby-kafka v0.4.x.
 - **Kafka 0.10:** Full support for the Producer and Consumer API in ruby-kafka v0.5.x. Note that you _must_ run version 0.10.1 or higher of Kafka due to limitations in 0.10.0.
-- **Kafka 0.11:** Everything that works with Kafka 0.10 should still work, but so far no features specific to Kafka 0.11 have been added.
-- **Kafka 0.11:** Everything that works with Kafka 0.10 should still work, but so far no features specific to Kafka 1.0 have been added.
+- **Kafka 0.11:** Full support for Producer API, limited support for Consumer API in ruby-kafka v0.7.x. New features in 0.11.x includes new Record Batch format, idempotent and transactional production. The missing feature is dirty reading of Consumer API.
+- **Kafka 1.0:** Everything that works with Kafka 0.11 should still work, but so far no features specific to Kafka 1.0 have been added.
 
 This library requires Ruby 2.1 or higher.
 
@@ -749,7 +749,7 @@ All notifications have `group_id` in the payload, referring to the Kafka consume
   * `key` is the message key.
   * `topic` is the topic that the message was consumed from.
   * `partition` is the topic partition that the message was consumed from.
-  * `offset` is the message's offset within the topic partition. 
+  * `offset` is the message's offset within the topic partition.
   * `offset_lag` is the number of messages within the topic partition that have not yet been consumed.
 
 * `start_process_message.consumer.kafka` is sent before `process_message.consumer.kafka`, and contains the same payload. It is delivered _before_ the message is processed, rather than _after_.
@@ -758,7 +758,7 @@ All notifications have `group_id` in the payload, referring to the Kafka consume
   * `message_count` is the number of messages in the batch.
   * `topic` is the topic that the message batch was consumed from.
   * `partition` is the topic partition that the message batch was consumed from.
-  * `highwater_mark_offset` is the message batch's highest offset within the topic partition. 
+  * `highwater_mark_offset` is the message batch's highest offset within the topic partition.
   * `offset_lag` is the number of messages within the topic partition that have not yet been consumed.
 
 * `start_process_batch.consumer.kafka` is sent before `process_batch.consumer.kafka`, and contains the same payload. It is delivered _before_ the batch is processed, rather than _after_.
@@ -767,21 +767,21 @@ All notifications have `group_id` in the payload, referring to the Kafka consume
   * `group_id` is the consumer group id.
 
 * `sync_group.consumer.kafka` is sent whenever a consumer is assigned topic partitions within a consumer group. It includes the following payload:
-  * `group_id` is the consumer group id.  
-  
+  * `group_id` is the consumer group id.
+
 * `leave_group.consumer.kafka` is sent whenever a consumer leaves a consumer group. It includes the following payload:
   * `group_id` is the consumer group id.
-  
+
 * `seek.consumer.kafka` is sent when a consumer first seeks to an offset. It includes the following payload:
   * `group_id` is the consumer group id.
   * `topic` is the topic we are seeking in.
   * `partition` is the partition we are seeking in.
   * `offset` is the offset we have seeked to.
-  
+
 * `heartbeat.consumer.kafka` is sent when a consumer group completes a heartbeat. It includes the following payload:
   * `group_id` is the consumer group id.
   * `topic_partitions` is a hash of { topic_name => array of assigned partition IDs }
-  
+
 #### Connection Notifications
 
 * `request.connection.kafka` is sent whenever a network request is sent to a Kafka broker. It includes the following payload:
@@ -908,7 +908,7 @@ can use:
 kafka = Kafka.new(["kafka1:9092"], ssl_ca_certs_from_system: true)
 ```
 
-This configures the store to look up CA certificates from the system default certificate store on an as needed basis. The location of the store can usually be determined by: 
+This configures the store to look up CA certificates from the system default certificate store on an as needed basis. The location of the store can usually be determined by:
 `OpenSSL::X509::DEFAULT_CERT_FILE`
 
 ##### Client Authentication
@@ -963,7 +963,7 @@ kafka = Kafka.new(
 ```
 
 ##### SCRAM
-Since 0.11 kafka supports [SCRAM](https://kafka.apache.org/documentation.html#security_sasl_scram). 
+Since 0.11 kafka supports [SCRAM](https://kafka.apache.org/documentation.html#security_sasl_scram).
 
 ```ruby
 kafka = Kafka.new(

data/lib/kafka/client.rb

@@ -618,7 +618,15 @@ module Kafka
     #
     # @return [Array<String>] the list of topic names.
     def topics
-      @cluster.list_topics
+      attempts = 0
+      begin
+        attempts += 1
+        @cluster.list_topics
+      rescue Kafka::ConnectionError
+        @cluster.mark_as_stale!
+        retry unless attempts > 1
+        raise
+      end
     end
 
     # Lists all consumer groups in the cluster

data/lib/kafka/cluster.rb

@@ -397,6 +397,7 @@ module Kafka
     end
 
     def random_broker
+      refresh_metadata_if_necessary!
       node_id = cluster_info.brokers.sample.node_id
       connect_to_broker(node_id)
     end

data/lib/kafka/consumer.rb

@@ -153,7 +153,8 @@ module Kafka
     def resume(topic, partition)
       pause_for(topic, partition).resume!
 
-      seek_to_next(topic, partition)
+      # During re-balancing we might have lost the paused partition. Check if partition is still in group before seek.
+      seek_to_next(topic, partition) if @group.assigned_to?(topic, partition)
     end
 
     # Whether the topic partition is currently paused.

data/lib/kafka/protocol/message.rb

@@ -102,6 +102,15 @@ module Kafka
         new(key: key, value: value, codec_id: codec_id, offset: offset, create_time: create_time)
       end
 
+      # Ensure the backward compatibility of Message format from Kafka 0.11.x
+      def is_control_record
+        false
+      end
+
+      def headers
+        {}
+      end
+
       private
 
       # Offsets may be relative with regards to wrapped message offset, but there are special cases.

data/lib/kafka/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Kafka
-  VERSION = "0.7.1"
+  VERSION = "0.7.2"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: ruby-kafka
 version: !ruby/object:Gem::Version
-  version: 0.7.1
+  version: 0.7.2
 platform: ruby
 authors:
 - Daniel Schierbeck
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2018-09-12 00:00:00.000000000 Z
+date: 2018-09-24 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: digest-crc