heller 0.0.3-java → 0.2.0-java

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: eb77b9b32f52f9f8cb4aefbdf3d27af37b402283
+  data.tar.gz: dcf3fbab3d4ee6cb31f0da70e98f18fe3994b2b6
+SHA512:
+  metadata.gz: 8a4b163c1f7523d3035cf73087b8d26dc366845b66c3c39845e9476fcb9d00b32944a7307a9e746deb833db722605290e2576322116b0a5ccbd65bb68027d6f1
+  data.tar.gz: 9658bb7e731c353d0269bcce8c150c7f47883d21bf0c592d937eaab50c641b0f064037dabe846698443d2acc7987995eb9084152aac60f2d5b06ed9bd4d5fa9d

data/README.md

@@ -0,0 +1,193 @@
+# Heller
+
+[![Build Status](https://travis-ci.org/mthssdrbrg/heller.svg?branch=master)](https://travis-ci.org/mthssdrbrg/heller)
+[![Coverage Status](https://coveralls.io/repos/mthssdrbrg/heller/badge.svg?branch=master)](https://coveralls.io/r/mthssdrbrg/heller?branch=master)
+
+Heller is a JRuby wrapper around the Kafka Producer and (Simple)Consumer
+APIs, much like [Mikka](https://github.com/iconara/mikka) is for Akka's Java API.
+
+The goal of Heller is to make the Producer and Consumer APIs of Kafka a bit more
+Rubyesque, and useful as building blocks for creating more advanced producer and
+consumer implementations.
+
+## Producer API
+
+```Heller::Producer``` is an extremely simple wrapper class around
+```kafka.javaapi.producer.Producer``` and provides some convenience in configuring the
+producer with more Rubyesque names for configuration parameters.
+
+All configuration parameters are supported and can be used in the following way:
+
+```ruby
+require 'heller'
+
+producer = Heller::Producer.new('localhost:9092,localhost:9093', {
+  :type => :async,
+  :serializer => 'kafka.serializer.StringEncoder',
+  :key_serializer => 'kafka.serializer.DefaultEncoder',
+  :partitioner => 'kafka.producer.DefaultPartitioner',
+  :compression => :gzip,
+  :num_retries => 5,
+  :retry_backoff => 1500,
+  :metadata_refresh_interval => 5000,
+  :batch_size => 2000,
+  :client_id => 'spec-client',
+  :request_timeout => 10000,
+  :buffer_limit => 100 * 100,
+  :buffer_timeout => 1000 * 100,
+  :enqueue_timeout => 1000,
+  :socket_buffer => 1024 * 1000,
+  :ack => -1
+})
+```
+
+Check the official [Kafka docs](http://kafka.apache.org/documentation.html#producerconfigs) for possible values for each parameter.
+
+To send messages you creates instances of ```Heller::Message``` and feed them to the
+```#push``` method of the producer:
+
+```ruby
+messages = 3.times.map { Heller::Message.new('test', 'my message!') }
+producer.push(messages)
+```
+
+Want to partition messages based on some key? Sure, no problem:
+
+```ruby
+messages = [0, 1, 2].map { |key| Heller::Message.new('test', "my message using #{key} as key!", key.to_s) }
+producer.push(messages)
+```
+
+## Consumer API
+
+```Heller::Consumer``` wraps ```kafka.javaapi.consumer.SimpleConsumer``` and provides
+basically the same methods, but with a bit more convenience (or at least I'd
+like to think so).
+
+A ```Consumer``` can be created in the following way:
+
+```ruby
+require 'heller'
+
+options = {
+  # 'generic' consumer options
+  :timeout => 5000,            # socket timeout
+  :buffer_size => 128 * 1024,  # socket buffer size
+  :client_id => 'my-consumer', # id of consumer
+  # fetch request related options
+  :max_wait => 4500,           # maximum time (ms) the consumer will wait for response of a request
+  :min_bytes => 1024           # minimum amount of bytes the server (broker) should return for a fetch request
+}
+
+consumer = Heller::Consumer.new('localhost:9092', options)
+```
+
+The options specified in the options hash are also described in the official
+[Kafka docs](http://kafka.apache.org/documentation.html#consumerconfigs), albeit they're described in the context of their high-level
+consumer.
+
+The consumer API exposes the following methods: ```#fetch```, ```#metadata```,
+```#offsets_before```, ```#earliest_offset``` and ```#latest_offset```, and
+their usage is described below.
+
+### Fetching messages
+
+```ruby
+topic = 'my-topic'
+partition = offset = 0
+
+response = consumer.fetch(Heller::FetchRequest.new(topic, partition, offset))
+
+if response.error? && (error_code = response.error(topic, partition)) != 0
+  puts "Got error #{Heller::Errors.error_for(error_code)}!"
+else
+  message_enumerator = response.messages(topic, partition)
+  message_enumerator.each do |offset, message_payload|
+    puts "#{offset}: #{message_payload}"
+  end
+end
+```
+
+See ```Heller::FetchResponse``` (and the related specs) for usage of other
+methods.
+
+It's also possible to pass an array of ```FetchRequest``` objects to ```#fetch```.
+
+```ruby
+requests = [0, 1, 2].map { |i| Heller::FetchRequest.new(topic, i, offset) }
+fetch_response = consumer.fetch(requests)
+```
+
+### Topic and partition metadata
+
+```kafka.javaapi.consumer.SimpleConsumer``` exposes a method called ```#topic_metadata```, which in Heller has been "renamed" to just ```#metadata```.
+
+```ruby
+topics = [1, 2, 3].map { |i| "my-topic-#{i}" }
+
+response = consumer.metadata(topics)
+
+response.each do |topic, partition_metadata|
+  puts "Got metadata for (#{topic}:#{partition_metadata.partition_id})"
+end
+
+leader = response.leader_for('my-topic-1', 0)
+puts "Leader for my-topic-1:0 is at #{leader.connection_string} (#{leader.zk_string})"
+
+isrs = response.isr_for('my-topic-1', 0) # also aliased as #in_sync_replicas_for
+isrs.each do |isr|
+  puts "An in-sync replica for my-topic-1:0 is at #{isr.connection_string} (#{isr.zk_string})"
+end
+```
+
+### Get offsets for topic-partition combinations
+
+```ruby
+# arguments = *[topic, partition, timestamp (ms), max number of offsets]
+request = Heller::OffsetRequest.new('my-topic', 0, Time.now.to_i * 1000, 10)
+response = consumer.offsets_before(request)
+
+if response.error? && (error_code = response.error('my-topic', 0)) != 0
+  puts "Got error #{Heller::Errors.error_for(error_code)}!"
+else
+  offsets = response.offsets('my-topic', 0)
+  puts "Got #{offsets.join(', ')} offsets for my-topic:0"
+end
+```
+
+```Heller::Consumer``` also exposes (as ```SimpleConsumer```) two convenience
+methods for retrieving the earliest / latest offset for a topic-partition
+combination.
+
+```ruby
+earliest_offset = consumer.earliest_offset('my-topic', 0)
+latest_offset = consumer.latest_offset('my-topic', 0)
+
+puts "Earliest available offset is #{earliest_offset}"
+puts "Latest available offset is #{latest_offset}"
+```
+
+## Status
+
+The project is currently under development, and I wouldn't really recommend it
+to be used in any form of production environment.
+There is still quite some work that needs to be done, especially for the Consumer API.
+The Producer API is more or less done, for the moment at least.
+
+It's getting there, though I'm mostly doing this during my spare time, which is
+sparse at times.
+
+## Copyright
+
+Copyright 2013-2015 Mathias Söderberg and contributors
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use
+this file except in compliance with the License. You may obtain a copy of the
+License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed
+under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
+CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.

data/lib/heller/configuration.rb

@@ -0,0 +1,41 @@
+# encoding: utf-8
+
+module Heller
+  class Configuration
+    def initialize(options={})
+      @configuration = merge_with_defaults(options)
+    end
+
+    def [](key)
+      @configuration[key.to_sym]
+    end
+
+    def to_java
+      kafka_config_class.new(to_properties)
+    end
+
+    protected
+
+    def defaults
+      {}
+    end
+
+    private
+
+    def merge_with_defaults(options)
+      options.each_with_object(defaults) do |(k, v), h|
+        h[k.to_sym] = v
+      end
+    end
+
+    def convert_key(key)
+      key_mappings.key?(key) ? key_mappings[key] : key.to_s.gsub('_', '.')
+    end
+
+    def to_properties
+      @configuration.each_with_object(Properties.new) do |(key, value), props|
+        props.put(convert_key(key), value.to_s)
+      end
+    end
+  end
+end

data/lib/heller/consumer.rb

@@ -1,59 +1,95 @@
+# encoding: utf-8
+
+require 'securerandom'
+
+
 module Heller
-	class Consumer < Kafka::Consumer::SimpleConsumer
-				
-		LATEST_OFFSET = -1.freeze
-		EARLIEST_OFFSET = -2.freeze
-
-		MAX_FETCH_SIZE = 1000000.freeze
-
-		def latest_offset(topic, partition)
-			single_offset(topic, partition, LATEST_OFFSET)
-		end
-
-		def earliest_offset(topic, partition)
-			single_offset(topic, partition, EARLIEST_OFFSET)
-		end
-
-		def fetch_request(topic, partition, offset, max_size)
-			Kafka::Api::FetchRequest.new(topic, partition, offset, max_size)
-		end
-
-		def consume(topic, partition, offset, max_size = MAX_FETCH_SIZE)
-			request = fetch_request(topic, partition, offset, max_size)
-
-			messages = fetch(request)
-			messages.to_a
-		end
-
-		def multi_fetch(topics_hash, max_size = MAX_FETCH_SIZE)
-			requests = topics_hash.map { |topic, hash| fetch_request(topic, hash[:partition], hash[:offset], max_size) }
-
-			response = multifetch(ArrayList.new(requests))
-			parse_multi_fetch_response(topics_hash.keys, response)
-		end
-
-		protected
-
-		def single_offset(topic, partition, time)
-			offsets = get_offsets_before(topic, partition, time, 1)
-			offsets = offsets.to_a
-
-			offsets.first unless offsets.empty?
-		end
-
-		def parse_multi_fetch_response(topics, response)
-			if response.respond_to?(:to_a)
-				response_array = topics.zip(response.to_a)
-
-				response_array.inject({}) do |response_hash, (topic, messages)| 
-					response_hash[topic] = messages.to_a
-					response_hash
-				end
-			else
-				puts 'response does not respond to #to_a'
-				puts "response is #{response.inspect}"
-				puts "topics were: #{topics.inspect}"
-			end
-		end
-	end
+  class Consumer
+    def initialize(connect_string, options = {})
+      @host, @port = connect_string.split(':')
+      options   = defaults.merge(options)
+      @consumer = create_consumer(options)
+      @build_options = options.select { |k, _| BUILD_OPTIONS.include?(k) }
+      @decoder  = Kafka::Serializer::StringDecoder.new(nil)
+    end
+
+    def client_id
+      @consumer.client_id
+    end
+
+    def fetch(fetch_requests, fetch_size = DEFAULT_FETCH_SIZE)
+      builder = create_builder(@build_options)
+      Array(fetch_requests).each do |request|
+        builder.add_fetch(request.topic, request.partition, request.offset, fetch_size)
+      end
+      raw_response = @consumer.fetch(builder.build)
+      FetchResponse.new(raw_response, @decoder)
+    end
+
+    def metadata(topics=[])
+      request = Kafka::JavaApi::TopicMetadataRequest.new(topics)
+      TopicMetadataResponse.new(@consumer.send(request))
+    end
+    alias_method :topic_metadata, :metadata
+
+    def offsets_before(offset_requests)
+      request_info = Array(offset_requests).each_with_object({}) do |request, memo|
+        topic_partition = Kafka::Common::TopicAndPartition.new(request.topic, request.partition)
+        partition_offset = Kafka::Api::PartitionOffsetRequestInfo.new(request.time.to_i, request.max_offsets)
+
+        memo[topic_partition] = partition_offset
+      end
+
+      request = Kafka::JavaApi::OffsetRequest.new(request_info, OffsetRequest.current_version, client_id)
+      OffsetResponse.new(@consumer.get_offsets_before(request))
+    end
+
+    def earliest_offset(topic, partition)
+      response = offsets_before(OffsetRequest.new(topic, partition, OffsetRequest.earliest_time))
+      response.offsets(topic, partition).first
+    end
+
+    def latest_offset(topic, partition)
+      response = offsets_before(OffsetRequest.new(topic, partition, OffsetRequest.latest_time))
+      response.offsets(topic, partition).last
+    end
+
+    def disconnect
+      @consumer.close
+    end
+    alias_method :close, :disconnect
+
+    private
+
+    DEFAULT_FETCH_SIZE = 1024 * 1024
+    BUILD_OPTIONS = [:client_id, :max_wait, :min_bytes].freeze
+
+    def defaults
+      {
+        timeout: 30 * 1000,
+        buffer_size: 64 * 1024,
+        client_id: generate_client_id
+      }
+    end
+
+    def generate_client_id
+      "heller-#{self.class.name.split('::').last.downcase}-#{SecureRandom.uuid}"
+    end
+
+    def create_consumer(options)
+      consumer_impl = options.delete(:consumer_impl) || Kafka::Consumer::SimpleConsumer
+      extra_options = options.values_at(:timeout, :buffer_size, :client_id)
+      consumer_impl.new(@host, @port.to_i, *extra_options)
+    end
+
+    def create_builder(options)
+      builder = Kafka::Api::FetchRequestBuilder.new
+
+      BUILD_OPTIONS.each do |symbol|
+        builder.send(symbol, options[symbol]) if options[symbol]
+      end
+
+      builder
+    end
+  end
 end

data/lib/heller/consumer_configuration.rb

@@ -0,0 +1,38 @@
+# encoding: utf-8
+
+module Heller
+  class ConsumerConfiguration < Configuration
+
+    protected
+
+    def key_mappings
+      @key_mappings ||= {
+        auto_commit: 'auto.commit.enable',
+        auto_commit_interval: 'auto.commit.interval.ms',
+        auto_reset_offset: 'auto.offset.reset',
+        client_id: 'client.id',
+        consumer_id: 'consumer.id',
+        fetch_message_max_bytes: 'fetch.message.max.bytes',
+        fetch_min_bytes: 'fetch.min.bytes',
+        fetch_max_wait: 'fetch.wait.max.ms',
+        group_id: 'group.id',
+        num_fetchers: 'num.consumer.fetchers',
+        max_queued_message_chunks: 'queued.max.message.chunks',
+        receive_buffer: 'socket.receive.buffer.bytes',
+        rebalance_retries: 'rebalance.max.retries',
+        rebalance_retry_backoff: 'rebalance.backoff.ms',
+        refresh_leader_backoff: 'refresh.leader.backoff.ms',
+        socket_timeout: 'socket.timeout.ms',
+        timeout: 'consumer.timeout.ms',
+        zk_connect: 'zookeeper.connect',
+        zk_session_timeout: 'zookeeper.session.timeout.ms',
+        zk_connection_timeout: 'zookeeper.connection.timeout.ms',
+        zk_sync_time: 'zookeeper.sync.time.ms',
+      }.freeze
+    end
+
+    def kafka_config_class
+      Kafka::Consumer::ConsumerConfig
+    end
+  end
+end

data/lib/heller/errors.rb

@@ -0,0 +1,9 @@
+# encoding: utf-8
+
+module Heller
+  Errors = Kafka::Errors::ErrorMapping
+
+  class Errors
+    self.singleton_class.send(:alias_method, :error_for, :exception_for)
+  end
+end

data/lib/heller/fetch_request.rb

@@ -0,0 +1,11 @@
+# encoding: utf-8
+
+module Heller
+  class FetchRequest
+    attr_reader :topic, :partition, :offset
+
+    def initialize(*args)
+      @topic, @partition, @offset = *args
+    end
+  end
+end

data/lib/heller/fetch_response.rb

@@ -0,0 +1,33 @@
+# encoding: utf-8
+
+module Heller
+  class FetchResponse
+    def initialize(underlying, decoder)
+      @underlying, @decoder = underlying, decoder
+    end
+
+    def error?
+      @underlying.has_error?
+    end
+
+    def error(topic, partition)
+      convert_error { @underlying.error_code(topic, partition) }
+    end
+
+    def messages(topic, partition)
+      convert_error { MessageSetEnumerator.new(@underlying.message_set(topic, partition), @decoder) }
+    end
+
+    def high_watermark(topic, partition)
+      convert_error { @underlying.high_watermark(topic, partition) }
+    end
+
+    private
+
+    def convert_error
+      yield
+    rescue IllegalArgumentException => e
+      raise NoSuchTopicPartitionCombinationError, e.message, e.backtrace
+    end
+  end
+end

data/lib/heller/message.rb

@@ -0,0 +1,9 @@
+# encoding: utf-8
+
+module Heller
+  class Message < Kafka::Producer::KeyedMessage
+    def initialize(topic, message, key = nil)
+      super(topic, key, message)
+    end
+  end
+end

data/lib/heller/message_set_enumerator.rb

@@ -0,0 +1,35 @@
+# encoding: utf-8
+
+module Heller
+  class MessageSetEnumerator
+    include Enumerable
+
+    def initialize(message_set, decoder)
+      @iterator, @decoder = message_set.iterator, decoder
+    end
+
+    def each
+      loop do
+        yield self.next
+      end
+    end
+
+    def next
+      if @iterator.has_next?
+        item = @iterator.next
+        offset, payload = item.offset, item.message.payload
+        [offset, decode(payload)]
+      else
+        raise StopIteration
+      end
+    end
+
+    private
+
+    def decode(payload)
+      bytes = Java::byte[payload.limit].new
+      payload.get(bytes)
+      @decoder.from_bytes(bytes)
+    end
+  end
+end

data/lib/heller/offset_request.rb

@@ -0,0 +1,23 @@
+# encoding: utf-8
+
+module Heller
+  class OffsetRequest
+    attr_reader :topic, :partition, :time, :max_offsets
+
+    def self.latest_time
+      Kafka::Api::OffsetRequest.latest_time
+    end
+
+    def self.earliest_time
+      Kafka::Api::OffsetRequest.earliest_time
+    end
+
+    def self.current_version
+      Kafka::Api::OffsetRequest.current_version
+    end
+
+    def initialize(topic, partition, time, offsets = 1)
+      @topic, @partition, @time, @max_offsets = topic, partition, time, offsets
+    end
+  end
+end

data/lib/heller/offset_response.rb

@@ -0,0 +1,29 @@
+# encoding: utf-8
+
+module Heller
+  class OffsetResponse
+    def initialize(underlying)
+      @underlying = underlying
+    end
+
+    def offsets(topic, partition)
+      convert_error { @underlying.offsets(topic, partition).to_a }
+    end
+
+    def error?
+      @underlying.has_error?
+    end
+
+    def error(topic, partition)
+      convert_error { @underlying.error_code(topic, partition) }
+    end
+
+    private
+
+    def convert_error
+      yield
+    rescue NoSuchElementException => e
+      raise NoSuchTopicPartitionCombinationError, e.message, e.backtrace
+    end
+  end
+end

data/lib/heller/producer.rb

@@ -1,39 +1,25 @@
-module Heller
-	class Producer < Kafka::Producer::Producer
-
-		attr_reader :configuration
-
-		def initialize(zk_connect, options = {})
-			options.merge!({
-				'zk.connect' => zk_connect
-			})
-
-			@configuration = Kafka::Producer::ProducerConfig.new(hash_to_properties(options))
-			super @configuration
-		end
-
-		def produce(topic_mappings)
-			producer_data = topic_mappings.map do |topic, hash|
-				if hash[:key]
-					Kafka::Producer::ProducerData.new(topic, hash[:key], hash[:messages])
-				else
-					Kafka::Producer::ProducerData.new(topic, hash[:messages])
-				end
-			end
+# encoding: utf-8
 
-			send(producer_data)
-		end
-
-		protected
-
-		def hash_to_properties(options)
-			properties = java.util.Properties.new
-
-			options.each do |key, value|
-				properties.put(key.to_s, value.to_s)
-			end
-
-			properties
-		end
-	end
-end
+module Heller
+  class Producer
+    def initialize(broker_list, options = {})
+      @producer = create_producer(options.merge(brokers: broker_list))
+    end
+
+    def push(messages)
+      @producer.send(ArrayList.new(Array(messages)))
+    end
+
+    def disconnect
+      @producer.close
+    end
+    alias_method :close, :disconnect
+
+    private
+
+    def create_producer(options)
+      producer_impl = options.delete(:producer_impl) || Kafka::Producer::Producer
+      producer_impl.new(ProducerConfiguration.new(options).to_java)
+    end
+  end
+end