kafka 0.5.1 → 0.5.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a6970af4b6ef931eb091068ee75220a2ce7a3f79a3afa7abd0dc40c9b6f88a18
-  data.tar.gz: cdda5ed68a3f9310fe6ce33930d3ebff71f5d9c53b735c2412e6a72fce9ba546
+  metadata.gz: ea31d6c027c3e63e1b9933bd8f130bf2d216c3e96e09ea7c583c8d15e0f0981d
+  data.tar.gz: 2c55aea758996dd97498d8fc92abd52bdb680311113cccf3afcbc26b2fb3ef46
 SHA512:
-  metadata.gz: 1cd6d75ccbb42e8d37d763f542c5c267fae092bb26558057f2f3b04de92d27021b4a57361ccf12ea40a8ebfcc234a651b27c71341ecb57327e9cbd32c486105f
-  data.tar.gz: 7cc3f4d3b70f11b20af2ed8d6fce065d8abdf5a53c5ab60bb39cfbb59c1a306b6eb876337bc2b7370bd83377934942e24868c46db5b511766a5f7296f2bea7bf
+  metadata.gz: 423144f6caf3b753e61aaa8db147139523eb5e0eb2f72948bc83a52642e1f7abcdc9dc36d7323cdf755d38d71293a55f8730a659c572751ef29ebb9107e5e45b
+  data.tar.gz: d2d78e83b8adf7da5fab16e0c288d68f70b603dec02c1671fee4359ff9af12ce3f08b8d8198eeee3eb16a5e1c349faf5060c78a377742a5ab4289e2b533638ef

data/CHANGELOG.md

@@ -1,5 +1,16 @@
 ## Unreleased
 
+## 0.5.2 / 2020-01-27
+
+* Fixes DeliveryReport#error? and DeliveryReport#successful? being swapped
+* Fixes Kafka::FFI::Client#cluster_id being incorrectly bound
+* Fixes naming issues in Message#headers and Message#detach_headers
+* Fixes naming issue in Config#set_ssl_cert
+* Fixes passing nil for a Kafka::FFI::Opaque
+* Adds all current RD_KAFKA_RESP_ERR_ constants
+* Adds Kafka::QueueFullError for when producer queue is full
+* Adds bindings for built in partitioners
+
 ## 0.5.1 / 2020-01-22
 
 * Kafka::Consumer now uses poll_set_consumer instead of a Poller thread

data/README.md

@@ -2,12 +2,17 @@
 
 [![Build Status](https://travis-ci.com/deadmanssnitch/kafka.svg?branch=master)](https://travis-ci.com/deadmanssnitch/kafka)
 [![Gem Version](https://badge.fury.io/rb/kafka.svg)](https://badge.fury.io/rb/kafka)
+[![Documentation](https://img.shields.io/badge/-Documentation-success)](https://deadmanssnitch.com/opensource/kafka/docs/)
 
-The kafka gem provides a general producer and consumer for
-[Apache Kafka](https://kafka.apache.org) using bindings to the official
-[C client librdkafka](https://github.com/edenhill/librdkafka).  The `Kafka::FFI`
-module implements an object oriented mapping to most of the librdkafka API,
-making it easier and safer to use than calling functions directly.
+Kafka provides a Ruby client for [Apache Kafka](https://kafka.apache.org) that
+leverages [librdkafka](https://github.com/edenhill/librdkafka) for its
+performance and general correctness.
+
+## Features
+- Thread safe Producer with sync and async delivery reporting
+- High-level balanced Consumer
+- Admin client
+- Object oriented librdkafka mappings for easy custom implementations
 
 ## ⚠️ Project Status: Beta ⚠️
 
@@ -24,6 +29,8 @@ against your application and traffic, implement missing functions (see
 `rake ffi:missing`), work with the API and make suggestions for improvements.
 All help is wanted and appreciated.
 
+Kafka is currently in production usage at [Dead Man's Snitch](https://deadmanssnitch.com).
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -40,15 +47,15 @@ Or install it yourself as:
 
     $ gem install kafka
 
-## Usage
+## Getting Started
 
 For more examples see [the examples directory](examples/).
 
 For a detailed introduction on librdkafka which would be useful when working
-with `Kafka::FFI` directly, see 
+with `Kafka::FFI` directly, see
 [the librdkafka documentation](https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md).
 
-### Sending Message to a Topic
+### Sending a Message to a Topic
 
 ```ruby
 require "kafka"
@@ -61,8 +68,17 @@ event = { time: Time.now, status: "success" }
 result = producer.produce("events", event.to_json)
 
 # Wait for the delivery to confirm that publishing was successful.
-# result.wait
-# result.successful?
+result.wait
+result.successful?
+
+# Provide a callback to be called when the delivery status is ready.
+producer.produce("events", event.to_json) do |result|
+  StatsD.increment("kafka.total")
+  
+  if result.error?
+    StatsD.increment("kafka.errors")
+  end
+end
 ```
 
 ### Consuming Messages from a Topic
@@ -144,11 +160,25 @@ amount of time we have available to invest. Embracing memory management and
 building clean separations between layers should reduce the burden to implement
 new bindings as the rules and responsibilities of each layer are clear.
 
+## Thread / Fork Safety
+
+The `Producer` is thread safe for publishing messages but should only be closed
+from a single thread. While the `Consumer` is thread safe for calls to `#poll`
+only one message can be in flight at a time, causing the threads to serialize.
+Instead, create a single consumer for each thread.
+
+Kafka _is not_ `fork` safe. Make sure to close any Producers or Consumer before
+forking and rebuild them after forking the new process.
+
+## Compatibility
+
+Kafka requires Ruby 2.5+ and is tested against Ruby 2.5+ and Kafka 2.1+.
+
 ## Development
 
-To get started with development make sure to have docker, docker-compose, and
-[kafkacat](https://github.com/edenhill/kafkacat) installed as they make getting
-up to speed easier.
+To get started with development make sure to have `docker`, `docker-compose`, and
+[`kafkacat`](https://github.com/edenhill/kafkacat) installed as they make getting
+up to speed easier. Some rake tasks depend on `ctags`.
 
 Before running the test, start a Kafka broker instance
 
@@ -175,9 +205,11 @@ the [code of conduct](https://github.com/deadmanssnitch/kafka/blob/master/CODE_O
 
 ## License
 
-The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+The gem is available as open source under the terms of the
+[MIT License](https://opensource.org/licenses/MIT).
 
 ## Code of Conduct
 
-Everyone interacting in the Kafka project's codebases and issue trackers are expected to follow the
+Everyone interacting in the Kafka project's codebases and issue trackers are
+expected to follow the
 [code of conduct](https://github.com/deadmanssnitch/kafka/blob/master/CODE_OF_CONDUCT.md).

data/Rakefile

@@ -15,14 +15,13 @@ end
 task default: [:ext, :spec]
 
 namespace :ffi do
-  desc "Lists the librdkafka functions that have not been implemented in Kafka::FFI"
-  task :missing do
-    require_relative "lib/kafka/version"
+  require_relative "lib/kafka/version"
 
-    require "uri"
-    require "net/http"
-    require "tempfile"
+  require "uri"
+  require "net/http"
+  require "tempfile"
 
+  def header
     header = Tempfile.new(["rdkafka", ".h"])
 
     # Fetch the header for the pinned version of librdkafka. rdkafka.h contains
@@ -32,7 +31,14 @@ namespace :ffi do
     header.write(resp)
     header.close
 
-    all = `ctags -x --sort=yes --kinds-C=pf #{header.path} | awk '{ print $1 }'`
+    at_exit { header.unlink }
+
+    header.path
+  end
+
+  desc "Lists the librdkafka functions that have not been implemented in Kafka::FFI"
+  task :missing do
+    all = `ctags -x --sort=yes --kinds-C=pf #{header} | awk '{ print $1 }'`
     all = all.split("\n")
 
     ffi_path = File.expand_path("lib/kafka/ffi.rb", __dir__)
@@ -41,8 +47,6 @@ namespace :ffi do
 
     missing = all - implemented
     puts missing
-  ensure
-    header.unlink
   end
 
   desc "Prints the list of implemented librdkafka functions"
@@ -50,6 +54,29 @@ namespace :ffi do
     ffi_path = File.expand_path("lib/kafka/ffi.rb", __dir__)
     puts `grep -o -h -P '^\\s+attach_function\\s+:\\Krd_kafka_\\w+' #{ffi_path} | sort`
   end
+
+  namespace :sync do
+    desc "Update ffi.rb with all errors defined in rdkafka.h"
+    task :errors do
+      ffi_path = File.expand_path("lib/kafka/ffi.rb", __dir__)
+
+      cmd = [
+        # Find all of the enumerator types in the header
+        "ctags -x --sort=no --kinds-C=e #{header}",
+
+        # Reduce it to just RD_KAFKA_RESP_ERR_* and their values
+        "grep -o -P 'RD_KAFKA_RESP_ERR_\\w+ = -?\\d+'",
+
+        # Add spacing to the constants so they line up correctly.
+        "sed -e 's/^/    /'",
+
+        # Delete any existing error constants then append the generated result.
+        "sed -e '/^\\s\\+RD_KAFKA_RESP_ERR_.\\+=.\\+/d' -e '/Response Errors/r /dev/stdin' #{ffi_path}",
+      ].join(" | ")
+
+      File.write(ffi_path, `#{cmd}`, mode: "w")
+    end
+  end
 end
 
 namespace :kafka do
@@ -64,6 +91,7 @@ namespace :kafka do
 
   desc "Shutdown the development Kafka instance"
   task :down do
-    sh "docker-compose -p ruby_kafka_dev down"
+    compose = Dir["spec/support/kafka-*.yml"].max
+    sh "docker-compose -p ruby_kafka_dev -f #{compose} down"
   end
 end

data/lib/kafka/consumer.rb

@@ -54,7 +54,7 @@ module Kafka
       @client.consumer_poll(timeout, &block)
     end
 
-    # @param msg [Consumer::Message]
+    # @param msg [Kafka::FFI::Message]
     def commit(msg, async: false)
       list = Kafka::FFI::TopicPartitionList.new(1)
 
@@ -66,7 +66,7 @@ module Kafka
       list.destroy
     end
 
-    # Gracefully shutdown the consumer and it's connections.
+    # Gracefully shutdown the consumer and its connections.
     #
     # @note After calling #close it is unsafe to call any other method on the
     #   Consumer.

data/lib/kafka/error.rb

@@ -9,6 +9,26 @@ module Kafka
   # @see rdkafka.h RD_KAFKA_RESP_ERR_*
   # @see rdkafka.h rd_kafka_resp_err_t
   class ::Kafka::ResponseError < Error
+    def self.new(code, message = nil)
+      klass =
+        case code
+        when Kafka::FFI::RD_KAFKA_RESP_ERR__QUEUE_FULL
+          QueueFullError
+        else
+          ResponseError
+        end
+
+      error = klass.allocate
+      error.send(:initialize, code, message)
+      error
+    end
+
+    # exception is called instead of `new` when using the form:
+    #   raise Kafka::ResponseError, code
+    def self.exception(code)
+      new(code)
+    end
+
     # @attr code [Integer] Error code as defined by librdkafka.
     attr_reader :code
 
@@ -41,4 +61,10 @@ module Kafka
       @message || ::Kafka::FFI.rd_kafka_err2str(@code)
     end
   end
+
+  # QueueFullError is raised when producing messages and the queue of messages
+  # pending delivery to topics has reached it's size limit.
+  #
+  # @see Config option `queue.buffering.max.messages`
+  class QueueFullError < ResponseError; end
 end

data/lib/kafka/ffi.rb

@@ -59,12 +59,145 @@ module Kafka
     ]
 
     # Response Errors
+    RD_KAFKA_RESP_ERR__BEGIN = -200
+    RD_KAFKA_RESP_ERR__BAD_MSG = -199
+    RD_KAFKA_RESP_ERR__BAD_COMPRESSION = -198
+    RD_KAFKA_RESP_ERR__DESTROY = -197
+    RD_KAFKA_RESP_ERR__FAIL = -196
+    RD_KAFKA_RESP_ERR__TRANSPORT = -195
+    RD_KAFKA_RESP_ERR__CRIT_SYS_RESOURCE = -194
+    RD_KAFKA_RESP_ERR__RESOLVE = -193
+    RD_KAFKA_RESP_ERR__MSG_TIMED_OUT = -192
+    RD_KAFKA_RESP_ERR__PARTITION_EOF = -191
+    RD_KAFKA_RESP_ERR__UNKNOWN_PARTITION = -190
+    RD_KAFKA_RESP_ERR__FS = -189
+    RD_KAFKA_RESP_ERR__UNKNOWN_TOPIC = -188
+    RD_KAFKA_RESP_ERR__ALL_BROKERS_DOWN = -187
+    RD_KAFKA_RESP_ERR__INVALID_ARG = -186
     RD_KAFKA_RESP_ERR__TIMED_OUT = -185
+    RD_KAFKA_RESP_ERR__QUEUE_FULL = -184
+    RD_KAFKA_RESP_ERR__ISR_INSUFF = -183
+    RD_KAFKA_RESP_ERR__NODE_UPDATE = -182
+    RD_KAFKA_RESP_ERR__SSL = -181
+    RD_KAFKA_RESP_ERR__WAIT_COORD = -180
+    RD_KAFKA_RESP_ERR__UNKNOWN_GROUP = -179
+    RD_KAFKA_RESP_ERR__IN_PROGRESS = -178
+    RD_KAFKA_RESP_ERR__PREV_IN_PROGRESS = -177
+    RD_KAFKA_RESP_ERR__EXISTING_SUBSCRIPTION = -176
     RD_KAFKA_RESP_ERR__ASSIGN_PARTITIONS = -175
     RD_KAFKA_RESP_ERR__REVOKE_PARTITIONS = -174
+    RD_KAFKA_RESP_ERR__CONFLICT = -173
+    RD_KAFKA_RESP_ERR__STATE = -172
+    RD_KAFKA_RESP_ERR__UNKNOWN_PROTOCOL = -171
+    RD_KAFKA_RESP_ERR__NOT_IMPLEMENTED = -170
+    RD_KAFKA_RESP_ERR__AUTHENTICATION = -169
     RD_KAFKA_RESP_ERR__NO_OFFSET = -168
+    RD_KAFKA_RESP_ERR__OUTDATED = -167
+    RD_KAFKA_RESP_ERR__TIMED_OUT_QUEUE = -166
+    RD_KAFKA_RESP_ERR__UNSUPPORTED_FEATURE = -165
+    RD_KAFKA_RESP_ERR__WAIT_CACHE = -164
+    RD_KAFKA_RESP_ERR__INTR = -163
+    RD_KAFKA_RESP_ERR__KEY_SERIALIZATION = -162
+    RD_KAFKA_RESP_ERR__VALUE_SERIALIZATION = -161
+    RD_KAFKA_RESP_ERR__KEY_DESERIALIZATION = -160
+    RD_KAFKA_RESP_ERR__VALUE_DESERIALIZATION = -159
+    RD_KAFKA_RESP_ERR__PARTIAL = -158
+    RD_KAFKA_RESP_ERR__READ_ONLY = -157
     RD_KAFKA_RESP_ERR__NOENT = -156
+    RD_KAFKA_RESP_ERR__UNDERFLOW = -155
+    RD_KAFKA_RESP_ERR__INVALID_TYPE = -154
+    RD_KAFKA_RESP_ERR__RETRY = -153
+    RD_KAFKA_RESP_ERR__PURGE_QUEUE = -152
+    RD_KAFKA_RESP_ERR__PURGE_INFLIGHT = -151
     RD_KAFKA_RESP_ERR__FATAL = -150
+    RD_KAFKA_RESP_ERR__INCONSISTENT = -149
+    RD_KAFKA_RESP_ERR__GAPLESS_GUARANTEE = -148
+    RD_KAFKA_RESP_ERR__MAX_POLL_EXCEEDED = -147
+    RD_KAFKA_RESP_ERR__UNKNOWN_BROKER = -146
+    RD_KAFKA_RESP_ERR__END = -100
+    RD_KAFKA_RESP_ERR_UNKNOWN = -1
+    RD_KAFKA_RESP_ERR_NO_ERROR = 0
+    RD_KAFKA_RESP_ERR_OFFSET_OUT_OF_RANGE = 1
+    RD_KAFKA_RESP_ERR_INVALID_MSG = 2
+    RD_KAFKA_RESP_ERR_UNKNOWN_TOPIC_OR_PART = 3
+    RD_KAFKA_RESP_ERR_INVALID_MSG_SIZE = 4
+    RD_KAFKA_RESP_ERR_LEADER_NOT_AVAILABLE = 5
+    RD_KAFKA_RESP_ERR_NOT_LEADER_FOR_PARTITION = 6
+    RD_KAFKA_RESP_ERR_REQUEST_TIMED_OUT = 7
+    RD_KAFKA_RESP_ERR_BROKER_NOT_AVAILABLE = 8
+    RD_KAFKA_RESP_ERR_REPLICA_NOT_AVAILABLE = 9
+    RD_KAFKA_RESP_ERR_MSG_SIZE_TOO_LARGE = 10
+    RD_KAFKA_RESP_ERR_STALE_CTRL_EPOCH = 11
+    RD_KAFKA_RESP_ERR_OFFSET_METADATA_TOO_LARGE = 12
+    RD_KAFKA_RESP_ERR_NETWORK_EXCEPTION = 13
+    RD_KAFKA_RESP_ERR_COORDINATOR_LOAD_IN_PROGRESS = 14
+    RD_KAFKA_RESP_ERR_COORDINATOR_NOT_AVAILABLE = 15
+    RD_KAFKA_RESP_ERR_NOT_COORDINATOR = 16
+    RD_KAFKA_RESP_ERR_TOPIC_EXCEPTION = 17
+    RD_KAFKA_RESP_ERR_RECORD_LIST_TOO_LARGE = 18
+    RD_KAFKA_RESP_ERR_NOT_ENOUGH_REPLICAS = 19
+    RD_KAFKA_RESP_ERR_NOT_ENOUGH_REPLICAS_AFTER_APPEND = 20
+    RD_KAFKA_RESP_ERR_INVALID_REQUIRED_ACKS = 21
+    RD_KAFKA_RESP_ERR_ILLEGAL_GENERATION = 22
+    RD_KAFKA_RESP_ERR_INCONSISTENT_GROUP_PROTOCOL = 23
+    RD_KAFKA_RESP_ERR_INVALID_GROUP_ID = 24
+    RD_KAFKA_RESP_ERR_UNKNOWN_MEMBER_ID = 25
+    RD_KAFKA_RESP_ERR_INVALID_SESSION_TIMEOUT = 26
+    RD_KAFKA_RESP_ERR_REBALANCE_IN_PROGRESS = 27
+    RD_KAFKA_RESP_ERR_INVALID_COMMIT_OFFSET_SIZE = 28
+    RD_KAFKA_RESP_ERR_TOPIC_AUTHORIZATION_FAILED = 29
+    RD_KAFKA_RESP_ERR_GROUP_AUTHORIZATION_FAILED = 30
+    RD_KAFKA_RESP_ERR_CLUSTER_AUTHORIZATION_FAILED = 31
+    RD_KAFKA_RESP_ERR_INVALID_TIMESTAMP = 32
+    RD_KAFKA_RESP_ERR_UNSUPPORTED_SASL_MECHANISM = 33
+    RD_KAFKA_RESP_ERR_ILLEGAL_SASL_STATE = 34
+    RD_KAFKA_RESP_ERR_UNSUPPORTED_VERSION = 35
+    RD_KAFKA_RESP_ERR_TOPIC_ALREADY_EXISTS = 36
+    RD_KAFKA_RESP_ERR_INVALID_PARTITIONS = 37
+    RD_KAFKA_RESP_ERR_INVALID_REPLICATION_FACTOR = 38
+    RD_KAFKA_RESP_ERR_INVALID_REPLICA_ASSIGNMENT = 39
+    RD_KAFKA_RESP_ERR_INVALID_CONFIG = 40
+    RD_KAFKA_RESP_ERR_NOT_CONTROLLER = 41
+    RD_KAFKA_RESP_ERR_INVALID_REQUEST = 42
+    RD_KAFKA_RESP_ERR_UNSUPPORTED_FOR_MESSAGE_FORMAT = 43
+    RD_KAFKA_RESP_ERR_POLICY_VIOLATION = 44
+    RD_KAFKA_RESP_ERR_OUT_OF_ORDER_SEQUENCE_NUMBER = 45
+    RD_KAFKA_RESP_ERR_DUPLICATE_SEQUENCE_NUMBER = 46
+    RD_KAFKA_RESP_ERR_INVALID_PRODUCER_EPOCH = 47
+    RD_KAFKA_RESP_ERR_INVALID_TXN_STATE = 48
+    RD_KAFKA_RESP_ERR_INVALID_PRODUCER_ID_MAPPING = 49
+    RD_KAFKA_RESP_ERR_INVALID_TRANSACTION_TIMEOUT = 50
+    RD_KAFKA_RESP_ERR_CONCURRENT_TRANSACTIONS = 51
+    RD_KAFKA_RESP_ERR_TRANSACTION_COORDINATOR_FENCED = 52
+    RD_KAFKA_RESP_ERR_TRANSACTIONAL_ID_AUTHORIZATION_FAILED = 53
+    RD_KAFKA_RESP_ERR_SECURITY_DISABLED = 54
+    RD_KAFKA_RESP_ERR_OPERATION_NOT_ATTEMPTED = 55
+    RD_KAFKA_RESP_ERR_KAFKA_STORAGE_ERROR = 56
+    RD_KAFKA_RESP_ERR_LOG_DIR_NOT_FOUND = 57
+    RD_KAFKA_RESP_ERR_SASL_AUTHENTICATION_FAILED = 58
+    RD_KAFKA_RESP_ERR_UNKNOWN_PRODUCER_ID = 59
+    RD_KAFKA_RESP_ERR_REASSIGNMENT_IN_PROGRESS = 60
+    RD_KAFKA_RESP_ERR_DELEGATION_TOKEN_AUTH_DISABLED = 61
+    RD_KAFKA_RESP_ERR_DELEGATION_TOKEN_NOT_FOUND = 62
+    RD_KAFKA_RESP_ERR_DELEGATION_TOKEN_OWNER_MISMATCH = 63
+    RD_KAFKA_RESP_ERR_DELEGATION_TOKEN_REQUEST_NOT_ALLOWED = 64
+    RD_KAFKA_RESP_ERR_DELEGATION_TOKEN_AUTHORIZATION_FAILED = 65
+    RD_KAFKA_RESP_ERR_DELEGATION_TOKEN_EXPIRED = 66
+    RD_KAFKA_RESP_ERR_INVALID_PRINCIPAL_TYPE = 67
+    RD_KAFKA_RESP_ERR_NON_EMPTY_GROUP = 68
+    RD_KAFKA_RESP_ERR_GROUP_ID_NOT_FOUND = 69
+    RD_KAFKA_RESP_ERR_FETCH_SESSION_ID_NOT_FOUND = 70
+    RD_KAFKA_RESP_ERR_INVALID_FETCH_SESSION_EPOCH = 71
+    RD_KAFKA_RESP_ERR_LISTENER_NOT_FOUND = 72
+    RD_KAFKA_RESP_ERR_TOPIC_DELETION_DISABLED = 73
+    RD_KAFKA_RESP_ERR_FENCED_LEADER_EPOCH = 74
+    RD_KAFKA_RESP_ERR_UNKNOWN_LEADER_EPOCH = 75
+    RD_KAFKA_RESP_ERR_UNSUPPORTED_COMPRESSION_TYPE = 76
+    RD_KAFKA_RESP_ERR_STALE_BROKER_EPOCH = 77
+    RD_KAFKA_RESP_ERR_OFFSET_NOT_AVAILABLE = 78
+    RD_KAFKA_RESP_ERR_MEMBER_ID_REQUIRED = 79
+    RD_KAFKA_RESP_ERR_PREFERRED_LEADER_NOT_AVAILABLE = 80
+    RD_KAFKA_RESP_ERR_GROUP_MAX_SIZE_REACHED = 81
 
     # @see rdkafka.h rd_kafka_resp_err_t
     enum :error_code, [
@@ -276,7 +409,7 @@ module Kafka
     attach_function :rd_kafka_type, [Client], :kafka_type
     attach_function :rd_kafka_name, [Client], :string
     attach_function :rd_kafka_memberid, [Client], :pointer
-    attach_function :rd_kafka_clusterid, [Client], :pointer
+    attach_function :rd_kafka_clusterid, [Client, :timeout_ms], :pointer, blocking: true
     attach_function :rd_kafka_controllerid, [Client, :timeout_ms], :broker_id, blocking: true
     attach_function :rd_kafka_default_topic_conf_dup, [Client], TopicConfig
     attach_function :rd_kafka_conf, [Client], Config
@@ -440,14 +573,14 @@ module Kafka
     attach_function :rd_kafka_consume_start, [Topic, :partition, :offset], :int
     attach_function :rd_kafka_consume_start_queue, [Topic, :partition, :offset, Queue], :int
     attach_function :rd_kafka_consume_stop, [Topic, :partition], :int
-    attach_function :rd_kafka_consume, [Topic, :partition, :timeout_ms], Message.by_ref
-    attach_function :rd_kafka_consume_batch, [Topic, :partition, :timeout_ms, :pointer, :size_t], :ssize_t
-    attach_function :rd_kafka_consume_callback, [Topic, :partition, :timeout_ms, :consume_cb, :pointer], :int
+    attach_function :rd_kafka_consume, [Topic, :partition, :timeout_ms], Message.by_ref, blocking: true
+    attach_function :rd_kafka_consume_batch, [Topic, :partition, :timeout_ms, :pointer, :size_t], :ssize_t, blocking: true
+    attach_function :rd_kafka_consume_callback, [Topic, :partition, :timeout_ms, :consume_cb, :pointer], :int, blocking: true
 
     ### Simple Consumer Queue API
-    attach_function :rd_kafka_consume_queue, [Queue, :timeout_ms], Message.by_ref
-    attach_function :rd_kafka_consume_batch_queue, [Queue, :timeout_ms, :pointer, :size_t], :ssize_t
-    attach_function :rd_kafka_consume_callback_queue, [Queue, :timeout_ms, :consume_cb, :pointer], :int
+    attach_function :rd_kafka_consume_queue, [Queue, :timeout_ms], Message.by_ref, blocking: true
+    attach_function :rd_kafka_consume_batch_queue, [Queue, :timeout_ms, :pointer, :size_t], :ssize_t, blocking: true
+    attach_function :rd_kafka_consume_callback_queue, [Queue, :timeout_ms, :consume_cb, :pointer], :int, blocking: true
 
     ### Simple Consumer Topic + Partition API
     attach_function :rd_kafka_offset_store, [Topic, :partition, :offset], :error_code
@@ -460,17 +593,25 @@ module Kafka
     attach_function :rd_kafka_flush, [Producer, :timeout_ms], :error_code, blocking: true
     attach_function :rd_kafka_purge, [Producer, :int], :error_code, blocking: true
 
+    ## Partitioners
+
+    attach_function :rd_kafka_msg_partitioner_random, [Topic, :string, :size_t, :int32, Opaque, Opaque], :partition
+    attach_function :rd_kafka_msg_partitioner_consistent, [Topic, :string, :size_t, :int32, Opaque, Opaque], :partition
+    attach_function :rd_kafka_msg_partitioner_consistent_random, [Topic, :string, :size_t, :int32, Opaque, Opaque], :partition
+    attach_function :rd_kafka_msg_partitioner_murmur2, [Topic, :string, :size_t, :int32, Opaque, Opaque], :partition
+    attach_function :rd_kafka_msg_partitioner_murmur2_random, [Topic, :string, :size_t, :int32, Opaque, Opaque], :partition
+
     # Metadata
-    attach_function :rd_kafka_metadata, [Client, :bool, Topic, :pointer, :timeout_ms], :error_code
+    attach_function :rd_kafka_metadata, [Client, :bool, Topic, :pointer, :timeout_ms], :error_code, blocking: true
     attach_function :rd_kafka_metadata_destroy, [Metadata.by_ref], :void
 
     # Group List
-    attach_function :rd_kafka_list_groups, [Client, :string, :pointer, :timeout_ms], :error_code
+    attach_function :rd_kafka_list_groups, [Client, :string, :pointer, :timeout_ms], :error_code, blocking: true
     attach_function :rd_kafka_group_list_destroy, [GroupList.by_ref], :void
 
     # Queue
     attach_function :rd_kafka_queue_new, [Client], Queue
-    attach_function :rd_kafka_queue_poll, [Queue, :timeout_ms], Event
+    attach_function :rd_kafka_queue_poll, [Queue, :timeout_ms], Event, blocking: true
     attach_function :rd_kafka_queue_get_main, [Client], Queue
     attach_function :rd_kafka_queue_get_consumer, [Consumer], Queue
     attach_function :rd_kafka_queue_get_partition, [Consumer, :topic, :partition], Queue

data/lib/kafka/ffi/client.rb

@@ -17,7 +17,7 @@ module Kafka::FFI
   #
   # @note Naming this is hard and librdkafka primarily just refers to it as "a
   #   handle" to an instance. It's more akin to an internal service and this
-  #   Client talks the API to that service.
+  #   Client talks the API of that service.
   class Client < OpaquePointer
     require "kafka/ffi/consumer"
     require "kafka/ffi/producer"
@@ -101,6 +101,9 @@ module Kafka::FFI
     def initialize(ptr)
       super(ptr)
 
+      # Caches Topics created on the first call to #topic below. Topics need to
+      # be destroyed before destroying the Client. We keep a set in the client
+      # so end users don't need to think about it.
       @topics = {}
     end
 
@@ -108,6 +111,8 @@ module Kafka::FFI
     #
     # @note The returned config is read-only and tied to the lifetime of the
     #   Client. Don't try to modify or destroy the config.
+    #
+    # @return [Config] Client's current config. Read-only.
     def config
       ::Kafka::FFI.rd_kafka_conf(self)
     end
@@ -122,17 +127,23 @@ module Kafka::FFI
     # Retrieves the Client's Cluster ID
     #
     # @note requires config `api.version.request` set to true
-    def cluster_id
-      id = ::Kafka::FFI.rd_kafka_clusterid(self)
-
-      if id.null?
+    #
+    # @param timeout [Integer] Maximum time to wait in milliseconds. Use 0 for
+    # non-bloack call that will return immediately if metadata is cached.
+    #
+    # @return [nil] Cluster ID not available
+    # @return [String] ID of the Cluster
+    def cluster_id(timeout: 1000)
+      ptr = ::Kafka::FFI.rd_kafka_clusterid(self, timeout)
+      if ptr.null?
         return nil
       end
 
-      id.read_string
-    ensure
-      if !id.null?
-        ::Kafka::FFI.rd_kafka_mem_free(self, id)
+      begin
+        ptr.read_string
+      ensure
+        # Documentation explicitly says that the string needs to be freed.
+        ::Kafka::FFI.rd_kafka_mem_free(self, ptr)
       end
     end
 
@@ -161,25 +172,23 @@ module Kafka::FFI
     #   Topic can only be configured at creation.
     #
     # @raise [Kafka::ResponseError] Error occurred creating the topic
-    # @raise [TopicAlreadyConfiguredError] Passed a config for a topic that has
-    #   already been configured.
+    # @raise [Kafka::FFI::TopicAlreadyConfiguredError] Passed a config for a
+    #   topic that has already been configured.
     #
     # @return [Topic] Topic instance
-    #   error.
     def topic(name, config = nil)
       topic = @topics[name]
       if topic
         if config
           # Make this an exception because it's probably a programmer error
-          # that _should_ only primarily happen during development due to
+          # that _should_ primarily happen during development due to
           # misunderstanding the semantics.
-          raise TopicAlreadyConfigured, "#{name} was already configured"
+          raise ::Kafka::FFI::TopicAlreadyConfiguredError, "#{name} was already configured"
         end
 
         return topic
       end
 
-      # @todo - Keep list of topics and manage their lifecycle?
       topic = ::Kafka::FFI.rd_kafka_topic_new(self, name, config)
       if topic.nil?
         raise ::Kafka::ResponseError, ::Kafka::FFI.rd_kafka_last_error

data/lib/kafka/ffi/config.rb

@@ -353,7 +353,7 @@ module Kafka::FFI
     # @raise [ConfigError] Certificate was not properly encoded or librdkafka
     #   was not compiled with SSL/TLS.
     def set_ssl_cert(cert_type, cert_enc, certificate)
-      error = ::MemoryPointer.new(:char, 512)
+      error = ::FFI::MemoryPointer.new(:char, 512)
 
       err = ::Kafka::FFI.rd_kafka_conf_set_ssl_cert(cert_type, cert_enc, certificate, certificate.bytesize, error, error.size)
       if err != :ok

data/lib/kafka/ffi/message.rb

@@ -106,7 +106,7 @@ module Kafka::FFI
     # @raise [Kafka::ResponseError] Error occurred parsing headers
     #
     # @return [nil] Message does not have any headers
-    # @return [Message::Headers] Set of headers
+    # @return [Message::Header] Set of headers
     def headers
       ptr = ::FFI::MemoryPointer.new(:pointer)
 
@@ -116,7 +116,7 @@ module Kafka::FFI
         if ptr.null?
           nil
         else
-          Message::Headers.new(ptr)
+          Message::Header.new(ptr)
         end
       when RD_KAFKA_RESP_ERR__NOENT
         # Messages does not have headers
@@ -136,7 +136,7 @@ module Kafka::FFI
     # @raise [Kafka::ResponseError] Error occurred parsing headers
     #
     # @return [nil] Message does not have any headers
-    # @return [Message::Headers] Set of headers
+    # @return [Message::Header] Set of headers
     def detach_headers
       ptr = ::FFI::MemoryPointer.new(:pointer)
 
@@ -146,7 +146,7 @@ module Kafka::FFI
         if ptr.null?
           nil
         else
-          Message::Headers.new(ptr)
+          Message::Header.new(ptr)
         end
       when RD_KAFKA_RESP_ERR__NOENT
         # Messages does not have headers

data/lib/kafka/ffi/opaque.rb

@@ -41,8 +41,14 @@ module Kafka::FFI
         @registry.delete(opaque.pointer.address)
       end
 
-      # @param value [Opaque]
+      # @param value [Opaque, nil]
+      #
+      # @return [FFI::Pointer] Pointer referencing the Opauqe value.
       def to_native(value, _ctx)
+        if value.nil?
+          return ::FFI::Pointer::NULL
+        end
+
         value.pointer
       end
 

data/lib/kafka/ffi/producer.rb

@@ -33,6 +33,10 @@ module Kafka::FFI
     #   which will be available as Message#opaque in callbacks. The application
     #   MUST call #free on the Opaque once the final callback has been
     #   triggered to avoid leaking memory.
+    #
+    # @raise [Kafka::QueueFullError] Number of messages queued for delivery has
+    #   exceeded capacity. See `queue.buffering.max.messages` config setting to
+    #   adjust.
     def produce(topic, payload, key: nil, partition: nil, headers: nil, timestamp: nil, opaque: nil)
       args = [
         # Ensure librdkafka copies the payload into its own memory since the

data/lib/kafka/producer/delivery_report.rb

@@ -59,9 +59,13 @@ class Kafka::Producer
       @done
     end
 
-    # @return [Boolean] Is the report for an error?
+    # Returns if the delivery errored
+    #
+    # @see #see
+    #
+    # @return [Boolean] True when the delivery failed with an error.
     def error?
-      error.nil?
+      received? && !successful?
     end
 
     # Returns if the delivery was successful
@@ -69,7 +73,7 @@ class Kafka::Producer
     # @return [Boolean] True when the report was delivered to the cluster
     #   successfully.
     def successful?
-      !error
+      received? && error.nil?
     end
 
     # @private
@@ -97,6 +101,8 @@ class Kafka::Producer
       if @callback
         @callback.call(self)
       end
+
+      nil
     end
 
     # Wait for a report to be received for the delivery from the cluster.

data/lib/kafka/version.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module Kafka
-  VERSION = "0.5.1"
+  VERSION = "0.5.2"
 
   LIBRDKAFKA_VERSION  = "1.3.0"
   LIBRDKAFKA_CHECKSUM = "465cab533ebc5b9ca8d97c90ab69e0093460665ebaf38623209cf343653c76d2"

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: kafka
 version: !ruby/object:Gem::Version
-  version: 0.5.1
+  version: 0.5.2
 platform: ruby
 authors:
 - Chris Gaffney
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-01-22 00:00:00.000000000 Z
+date: 2020-01-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: ffi