codeclimate-poseidon 0.0.8

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 0d4c27c567e99772b9a024ffe551b68dd81185bf
+  data.tar.gz: b384e7627ec29faadbc7582a69312050ffe693bf
+SHA512:
+  metadata.gz: aab82245bd26818cad1a361bfefa20619c39e4f882af4204c5f0f4e0f2b17535e86829595cbb9ae1ed63d9fa1bc8fb63b9e604124e336d74dcbc5579f55d0fe3
+  data.tar.gz: 1ab14df98fc583eb6d48df2986421dc9ec33fbf39a17bfbc64ccc0172dffa2a7a07cb6a4eac3e0880a6298634f365f62a33c5ddfaea696d09f4684d8d77025ae

data/.gitignore

@@ -0,0 +1,21 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+*.log
+*.log.*
+tags
+.rvmrc

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--format progress

data/.travis.yml

@@ -0,0 +1,14 @@
+laguage: ruby
+rvm:
+  - 1.9.3
+  - 2.0.0
+  - 2.1
+  - ruby-head
+  - jruby-19mode
+  - jruby-head
+  - rbx-2
+matrix:
+  allow_failures:
+    - rvm: ruby-head
+    - rvm: jruby-head
+    - rvm: rbx-2

data/.yardopts

@@ -0,0 +1,8 @@
+--markup-provider=redcarpet
+--markup=markdown
+--no-private
+--files LICENSE.txt
+--exclude '~$'
+--title 'Poseidon (Kafka library for Ruby)'
+--api public
+--readme README.md

data/CHANGES.md

@@ -0,0 +1,31 @@
+# 0.0.5
+
+* Add support for negative offsets. [GH-24]
+* Fix serious bug where we would send messages to the wrong partition. [GH-36] (Thanks @sclasen and @jorgeortiz85 for tracking this down.)
+* Better error message when we can't connect to a broker. [GH-42]
+* Handle broker rebalances. [GH-43]
+* PartitionConsumer: Block for messages by default. [GH-48]
+* Add a logger to help debug issues. [GH-51]
+* Add snappy support. [GH-57]
+* Allow `:none` value for `:compression_codec` option. [GH-72]
+* Allow request buffer to accept mixed encodings. [GH-74]
+
+# 0.0.4
+
+* Don't truncate UTF8 Messages [GH-18]
+* Gracefully handle truncated fetch reponses [GH-19]
+
+# 0.0.3
+
+* Better distribute messages across partitions.
+* Handle broken connections better.
+* Gracefully handle attempts to send an empty set of messages.
+
+# 0.0.2
+
+* Added ability to create a partitioner consumer for a topic+partition using topic metadata.
+* Added PartitionConsumer#offset to return offset of the last fetch
+
+# 0.0.1
+
+* Initial release

data/Gemfile

@@ -0,0 +1,13 @@
+source 'https://rubygems.org'
+
+gem 'rake'
+
+# Specify your gem's dependencies in poseidon.gemspec
+gemspec
+
+gem 'coveralls', require: false
+
+group :development do
+  gem 'github-markup', :platform => :ruby
+  gem 'redcarpet', :platform => :ruby
+end

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2013 Bob Potter
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,72 @@
+# Poseidon [![Build Status](https://travis-ci.org/bpot/poseidon.png?branch=master)](https://travis-ci.org/bpot/poseidon) [![Code Climate](https://codeclimate.com/github/bpot/poseidon.png)](https://codeclimate.com/github/bpot/poseidon)
+
+Poseidon is a Kafka client. Poseidon only supports the 0.8 API and above.
+
+**Until 1.0.0 this should be considered ALPHA software and not neccessarily production ready.**
+
+## Usage
+
+### API Documentation
+
+* [Latest release](http://rubydoc.info/gems/poseidon)
+* [Github master](http://rubydoc.info/github/bpot/poseidon)
+
+### Installing a Kafka broker locally
+
+Follow the [instructions](http://kafka.apache.org/documentation.html#quickstart) on the Kafka wiki to build Kafka 0.8 and get a test broker up and running.
+
+### Sending messages to Kafka
+
+```ruby
+require 'poseidon'
+
+producer = Poseidon::Producer.new(["localhost:9092"], "my_test_producer")
+
+messages = []
+messages << Poseidon::MessageToSend.new("topic1", "value1")
+messages << Poseidon::MessageToSend.new("topic2", "value2")
+producer.send_messages(messages)
+```
+
+More detailed [Poseidon::Producer](http://rubydoc.info/github/bpot/poseidon/Poseidon/Producer) documentation.
+
+### Fetching messages from Kafka
+
+```ruby
+require 'poseidon'
+
+consumer = Poseidon::PartitionConsumer.new("my_test_consumer", "localhost", 9092,
+                                            "topic1", 0, :earliest_offset)
+
+loop do
+  messages = consumer.fetch
+  messages.each do |m|
+    puts m.value
+  end
+end
+```
+
+More detailed [Poseidon::PartitionConsumer](http://rubydoc.info/github/bpot/poseidon/Poseidon/PartitionConsumer) documentation.
+
+### Using snappy compression
+
+To use snappy compression in your producers or consumers, install the [snappy](http://rubygems.org/gems/snappy) gem or simply add `gem 'snappy'` to your project's Gemfile.
+
+## Semantic Versioning
+
+This gem follows [SemVer](http://semver.org). In particular, the public API should not be considered stable and anything may change without warning until Version 1.0.0.  Additionally, for the purposes of the versioning the public API is everything documented in the [public API docs](http://rubydoc.info/github/bpot/poseidon).
+
+## Requirements
+
+* Ruby 1.9.3 or higher (1.9.2 and below not supported!!!)
+* Kafka 0.8 or higher
+
+## Integration Tests
+
+In order to run integration tests you must specify a `KAFKA_PATH` environment variable which points to a built Kafka installation.  To build Kafka locally follow the [instructions](http://kafka.apache.org/documentation.html#quickstart) provided by the project.
+
+    # cd ~/src/poseidon/
+    # bundle
+    # KAFKA_PATH=~/src/kafka bundle exec rake spec:all # run all unit and integration specs
+
+The poseidon test suite will take care of spinning up and down the broker(s) needed for the integration tests.

data/Rakefile

@@ -0,0 +1,20 @@
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new("spec:unit") do |t|
+  t.pattern = 'spec/unit/**/*_spec.rb'
+end
+
+RSpec::Core::RakeTask.new('spec:integration:simple') do |t|
+  t.pattern = 'spec/integration/simple/*_spec.rb'
+  t.rspec_opts = ["--fail-fast", "-f d"]
+end
+
+RSpec::Core::RakeTask.new('spec:integration:multiple_brokers') do |t|
+  t.pattern = 'spec/integration/multiple_brokers/*_spec.rb'
+  t.rspec_opts = ["--fail-fast", "-f d"]
+end
+
+task :spec => 'spec:unit'
+task 'spec:all' => ['spec:unit', 'spec:integration:simple', 'spec:integration:multiple_brokers']
+task :default => 'spec:unit'

data/TODO.md

@@ -0,0 +1,27 @@
+### 0.0.1
+  * Ensure that protocol errors are being handled correctly and not bubbling up
+  * More integration tests, replication, leader changes, etc. Investigate interesting cases in kafka's tests
+  * End-to-end integration specs
+    - In specs that test broker failure, verify that messages were actually sent/not sent with a consumer.
+ 
+  * AsyncProducer
+    - Implement a bounded queue, sending thread, etc
+  * Cleanup: extract protocol struct delegation to a module.
+  * When failing to send messages in sync producer, return messages that failed to send?
+
+### 0.0.2
+
+  * New Consumer/Consumer Enhancements
+    - Automatically partition work among consumers (zookeeper, redis, pluggable?)
+    - Handle case where the offset we're trying to read from no longer exists
+
+  * Snappy Compression
+    - snappy: c-ext, would like to avoid
+    - snappy_ffi: ffi interface, but needs to be updated (pre c-api) 
+        and has no specs, docs. Also linked to a c-ext version, two gems, etc..
+    - new snappy ffi library with specs, docs, etc. Shave that Yak!
+
+  * Benchmark/Profiling. KGIO?
+
+### 0.0.3 -- Targets Kafka 0.8.1
+  - Offset API

data/examples/consumer.rb

@@ -0,0 +1,18 @@
+$:.unshift File.expand_path(File.dirname(__FILE__) + '/../lib')
+require 'poseidon'
+
+producer = Poseidon::PartitionConsumer.new("example_consumer", "localhost", 9092,
+                                           "example", 0, :earliest_offset)
+
+loop do
+  begin
+    messages = producer.fetch
+    messages.each do |m|
+      puts "Received message: #{m.value}"
+    end
+  rescue Poseidon::Errors::UnknownTopicOrPartition
+    puts "Topic does not exist yet"
+  end
+
+  sleep 1
+end

data/examples/producer.rb

@@ -0,0 +1,9 @@
+$:.unshift File.expand_path(File.dirname(__FILE__) + '/../lib')
+require 'poseidon'
+
+producer = Poseidon::Producer.new(["localhost:9092"], "example_producer")
+
+loop do
+  producer.send_messages([Poseidon::MessageToSend.new("example", Time.now.to_s)])
+  sleep 1
+end

data/lib/poseidon.rb

@@ -0,0 +1,120 @@
+# Stdlib requires
+require 'socket'
+require 'zlib'
+require 'thread'
+require 'set'
+require 'logger'
+require 'stringio'
+
+# Top level Poseidon namespace
+#
+# @api public
+module Poseidon
+  # Posiedon exception namespace
+  module Errors
+    # @api private
+    class ProtocolError < StandardError; end
+
+    # Protocol Exceptions
+    #
+    # These are defined by the Poseidon wire format,
+    # they should be caught before being raised to users.
+    #
+    # @api private
+    class UnknownError < ProtocolError; end
+    # @api private
+    class OffsetOutOfRange < ProtocolError; end
+    # @api private
+    class InvalidMessage < ProtocolError; end
+    # @api private
+    class UnknownTopicOrPartition < ProtocolError; end
+    # @api private
+    class InvalidMessageSize < ProtocolError; end
+    # @api private
+    class LeaderNotAvailable < ProtocolError; end
+    # @api private
+    class NotLeaderForPartition < ProtocolError; end
+    # @api private
+    class RequestTimedOut < ProtocolError; end
+    # @api private
+    class BrokerNotAvailable < ProtocolError; end
+    # @api private
+    class ReplicaNotAvailable < ProtocolError; end
+    # @api private
+    class MessageSizeTooLarge < ProtocolError; end
+    # @api private
+    class UnrecognizedProtocolError < ProtocolError; end
+
+    # @api private
+    NO_ERROR_CODE = 0
+    # @api private
+    ERROR_CODES = {
+      -1 => UnknownError,
+      1 => OffsetOutOfRange,
+      2 => InvalidMessage,
+      3 => UnknownTopicOrPartition,
+      4 => InvalidMessageSize,
+      5 => LeaderNotAvailable,
+      6 => NotLeaderForPartition,
+      7 => RequestTimedOut,
+      8 => BrokerNotAvailable,
+      9 => ReplicaNotAvailable,
+      10 => MessageSizeTooLarge
+    }
+
+    # Raised when a custom partitioner tries to send
+    # a message to a partition that doesn't exist.
+    class InvalidPartitionError < StandardError; end
+
+    # Raised when we are unable to fetch metadata from
+    # any of the brokers.
+    class UnableToFetchMetadata < StandardError; end
+
+    # Raised when a messages checksum doesn't match
+    class ChecksumError < StandardError; end
+
+    # Raised when you try to send messages to a producer
+    # object that has been #shutdown
+    class ProducerShutdownError < StandardError; end
+  end
+
+  def self.logger
+    @logger ||= null_logger
+  end
+
+  def self.logger=(logger)
+    @logger = logger
+  end
+
+  private
+  def self.null_logger
+    devnull = RUBY_PLATFORM =~ /w32/ ? 'nul' : '/dev/null'
+    l = Logger.new(devnull)
+    l.level = Logger::INFO
+    l
+  end
+end
+
+# Public API
+require "poseidon/message_to_send"
+require "poseidon/producer"
+require "poseidon/fetched_message"
+require "poseidon/partition_consumer"
+
+# Poseidon!
+require "poseidon/message"
+require "poseidon/message_set"
+require "poseidon/topic_metadata"
+require "poseidon/protocol"
+
+require "poseidon/broker_pool"
+require "poseidon/cluster_metadata"
+require "poseidon/compression"
+require "poseidon/connection"
+require "poseidon/message_conductor"
+require "poseidon/messages_for_broker"
+require "poseidon/messages_to_send"
+require "poseidon/messages_to_send_batch"
+require "poseidon/producer_compression_config"
+require "poseidon/sync_producer"
+require "poseidon/version"

data/lib/poseidon/broker_pool.rb

@@ -0,0 +1,86 @@
+module Poseidon
+  # BrokerPool allows you to send api calls to the a brokers Connection.
+  #
+  # @api private
+  class BrokerPool
+    class UnknownBroker < StandardError; end
+
+    # @yieldparam [BrokerPool]
+    def self.open(client_id, seed_brokers, socket_timeout_ms, &block)
+      broker_pool = new(client_id, seed_brokers, socket_timeout_ms)
+
+      yield broker_pool
+    ensure
+      broker_pool.close
+    end
+
+    # @param [String] client_id
+    def initialize(client_id, seed_brokers, socket_timeout_ms)
+      @connections = {}
+      @brokers = {}
+      @client_id = client_id
+      @seed_brokers = seed_brokers
+      @socket_timeout_ms = socket_timeout_ms
+    end
+
+    def fetch_metadata(topics)
+      @seed_brokers.each do |broker|
+        if metadata = fetch_metadata_from_broker(broker, topics)
+          Poseidon.logger.debug { "Fetched metadata from #{broker}:\n" + metadata.to_s }
+          return metadata
+        end
+      end
+      raise Errors::UnableToFetchMetadata
+    end
+
+    # Update the brokers we know about
+    #
+    # TODO break connection when a brokers info changes?
+    #
+    # @param [Hash<Integer,Hash>] brokers
+    #   Hash of broker_id => { :host => host, :port => port }
+    def update_known_brokers(brokers)
+      @brokers.update(brokers)
+      nil
+    end
+
+    # Executes an api call on the connection
+    #
+    # @param [Integer] broker_id id of the broker we want to execute it on
+    # @param [Symbol] api_call
+    #   the api call we want to execute (:produce,:fetch,etc)
+    def execute_api_call(broker_id, api_call, *args)
+      connection(broker_id).send(api_call, *args)
+    end
+
+    # Closes all open connections to brokers
+    def close
+      @brokers.values(&:close)
+      @brokers = {}
+    end
+
+    alias_method :shutdown, :close
+
+    private
+    def fetch_metadata_from_broker(broker, topics)
+      host, port = broker.split(":")
+      Connection.open(host, port, @client_id, @socket_timeout_ms) do |connection|
+        connection.topic_metadata(topics)
+      end
+    rescue Connection::ConnectionFailedError
+      return nil
+    end
+
+    def connection(broker_id)
+      @connections[broker_id] ||= new_connection(broker_id)
+    end
+
+    def new_connection(broker_id)
+      info = @brokers[broker_id]
+      if info.nil?
+        raise UnknownBroker
+      end
+      Connection.new(info[:host], info[:port], @client_id, @socket_timeout_ms)
+    end
+  end
+end