ktl 1.0.0-java

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 57f0bf86bb51e0401585eaec3993a4fb971d00c6
+  data.tar.gz: 7270677f6a1e5d7550dc35cdd51b351ac293e4bd
+SHA512:
+  metadata.gz: 75986093b1f59323ff894d9400b2921a7e66a3da5a60acc4b2639a24118011fa52a7e6143cde031b751c6a68f9cb865c0bd0a43ac80589ab960b75706b8f02d6
+  data.tar.gz: 0ed21803a5f5b99cd7809a63307625cb27f85a970adef1e39be894da140d41c675e85e8243a14e57ac108a90cfc10cba5c25cb80b10382548bdf899759b3b803

data/LICENSE.txt

@@ -0,0 +1,12 @@
+Copyright (c) 2015, Burt AB
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

data/README.md

@@ -0,0 +1,106 @@
+# ktl
+
+`ktl`, short for "Kafka Control Tool", is a command-line tool that attempts to
+make the management of Kafka clusters that host a lot of topics and partitions
+easier.
+
+## Motivation
+
+The main motivation for creating `ktl` was to automate, as much as possible, the
+process of moving partition leadership between brokers when adding, removing or
+replacing brokers, as (unfortunately) the tools bundled with Kafka are rather
+manual and quite cumbersome to use when one host a lot (> 1000) of topics /
+partitions.
+
+I highly recommend that you read the [official documentation](https://kafka.apache.org/documentation.html#operations)
+regarding operating Kafka before reading further, as quite a lot of the
+terminology is explained in the documentation and not necessarily explained in
+any great depth in this README but merely mentioned.
+
+## Usage
+
+`ktl` supports a couple of different commands for common tasks such as
+performing preferred replica elections, shuffling partition leadership, checking
+consumer lag, creating topics, and so on.
+Most of the commands supports the usage of regular expressions for filtering
+which topics that will be acted upon.
+For example it's possible to perform preferred replica elections or reassign
+partitions for a subset of topics.
+
+```shell
+$ ktl --help
+Commands:
+  ktl cluster SUBCOMMAND ...ARGS   # Commands for managing cluster(s)
+  ktl consumer SUBCOMMAND ...ARGS  # Commands for managing consumers
+  ktl help [COMMAND]               # Describe available commands or one specific command
+  ktl topic SUBCOMMAND ...ARGS     # Commands for managing topics
+```
+
+### Partition reassignments
+
+A "**partition reassignment**" (or just "reassignment" for short) is basically a
+JSON document that describes which brokers that should be leader and/or replica
+for certain partitions.
+There are currently three subcommands of the `cluster` command that will generate
+reassignments: `migrate-broker`, `shuffle` and `decommission-broker`.
+
+When starting a new reassignment, `ktl` will write the reassignment JSON to a
+`reassign` znode under a `/ktl`prefix in ZooKeeper, so that it's possible to
+track the progress of a reassignment.
+
+The `shuffle` subcommand can either perform a random reassignment of partitions,
+or it can use [rendezvous hashing](http://en.wikipedia.org/wiki/Rendezvous_hashing),
+which will minimize the number of partitions that has to move between replicas
+when adding or removing brokers. There is also rack aware rendezvous hashing,
+which will ensure that data is properly replicated across racks.
+
+To start a random reassignment of the replica assignment of partitions matching
+`^test.*`, but leaving the rest alone:
+```shell
+$ ktl cluster shuffle '^test.*' -z localhost:2181/test
+```
+To do the same thing but using rendezvous hashing:
+```shell
+$ ktl cluster shuffle '^test.*' -R -z localhost:2181/test
+```
+To do the same thing using rack aware rendezvous hashing:
+```shell
+$ ktl cluster shuffle '^test.*' -a -z localhost:2181/test
+```
+
+#### Dealing with ZooKeeper's znode limit
+
+If the resulting reassignment JSON is greater than 1 MB (which is usually the
+hard limit for a single znode), `ktl` will split the reassignment into smaller
+slices and store the "overflow" in ZooKeeper under a `overflow` znode (in
+properly-sized slices) so that it's available for further invocations.
+
+The previously mentioned "reassignment" commands also take an optional `-l /
+--limit` parameter to limit the number of partitions that will be included in
+the reassignment, as it can be beneficial to perform a reassignment in steps
+rather than all at once, as moving replicas for a lot of partitions will slow
+down your producers and consumers.
+The default is however to reassign as many partitions as possible.
+
+Note that `ktl` will not wait for the reassignment to finish, but to proceed
+with the next slice of the reassignment is just a matter of running the same
+command again and it'll prompt you whether to continue you were you left off
+or generate a new reassignment.
+
+### Managing topics
+
+While the main motivation for `ktl` was to deal with partition reassignments,
+there are also some commands for managing topics, most of which merely wraps the
+`kafka-topics.sh` tool that is bundled with Kafka, but with a slightly different
+interface.
+
+The most notable difference is the `create` subcommand that has the ability to create
+a consistent hashing plan and assign it to the new topic during creation.
+
+```shell
+$ ktl topic create 'some.topic' --partitions 3 --replication_factor 3 --rack_aware_allocation -z localhost:2181/test
+```
+
+## Copyright
+
+© 2015 Burt AB, see LICENSE.txt (BSD 3-Clause).

data/bin/ktl

@@ -0,0 +1,8 @@
+#!/usr/bin/env ruby
+
+$: << File.expand_path('../../lib', __FILE__)
+
+require 'ktl'
+
+
+Ktl::Cli.start(ARGV)

data/lib/ext/kafka.rb

@@ -0,0 +1,145 @@
+# encoding: utf-8
+
+require 'kafka-jars'
+
+module Log4j
+  include_package 'org.apache.log4j'
+  java_import 'org.apache.log4j.Logger'
+
+  BasicConfigurator.configure
+  org.apache.log4j.Logger.root_logger.set_level(Level::ERROR)
+end
+
+module ZkClient
+  java_import 'org.I0Itec.zkclient.ZkClient'
+
+  module Exception
+    include_package 'org.I0Itec.zkclient.exception'
+  end
+end
+
+module Scala
+  java_import 'scala.Console'
+  java_import 'scala.Tuple2'
+  java_import 'scala.Option'
+
+  class Tuple2
+    alias_method :first, :_1
+    alias_method :last, :_2
+
+    def elements
+      [first, last]
+    end
+  end
+  Tuple = Tuple2
+
+  module Collection
+    include_package 'scala.collection'
+
+    module Mutable
+      include_package 'scala.collection.mutable'
+    end
+
+    module Immutable
+      include_package 'scala.collection.immutable'
+    end
+  end
+end
+
+class ScalaEnumerable
+  include Enumerable
+
+  def initialize(underlying)
+    @underlying = underlying
+  end
+
+  def each(&block)
+    @underlying.foreach(&block)
+  end
+end
+
+module Kafka
+  module Utils
+    include_package 'kafka.utils'
+
+    def self.new_zk_client(zk_connect, timeout=30_000)
+      ::ZkClient::ZkClient.new(zk_connect, timeout, timeout, ZKStringSerializer)
+    end
+
+    def self.get_partitions_for_topic(zk, topic)
+      topics = Scala::Collection::Immutable::List.from_array([topic].to_java)
+      partitions = zk.get_partitions_for_topics(topics)
+      partitions.get(topic).get
+    end
+
+    def self.delete_topic(zk, topic)
+      acl = Kafka::Utils::ZkUtils::DefaultAcls(false)
+      zk.create_persistent_path(ZkUtils.get_delete_topic_path(topic), '', acl)
+    end
+  end
+
+  module Api
+    include_package 'kafka.api'
+  end
+
+  module Cluster
+    include_package 'kafka.cluster'
+  end
+
+  module Admin
+    include_package 'kafka.admin'
+
+    TopicCommandOptions = TopicCommand::TopicCommandOptions
+
+    def self.to_topic_options(hash)
+      options = hash.flat_map do |key, value|
+        kafka_key = '--' + key.to_s.gsub('_', '-')
+        if value.is_a?(Hash)
+          value.map { |k, v| [kafka_key, [k, v].join('=')] }
+        elsif value.is_a?(Array)
+          value.map { |v| [kafka_key, v] }
+        else
+          [kafka_key, value].compact
+        end
+      end
+      TopicCommandOptions.new(options.flatten)
+    end
+
+    def self.preferred_replica(zk_client, topics_partitions)
+      PreferredReplicaLeaderElectionCommand.write_preferred_replica_election_data(zk_client, topics_partitions)
+    end
+
+    def self.assign_replicas_to_brokers(brokers, partitions, repl_factor, index=-1, partition=-1)
+      assignment = AdminUtils.assign_replicas_to_brokers(brokers, partitions.to_java(:int), repl_factor.to_java(:int), index.to_java(:int), partition.to_java(:int))
+      ScalaEnumerable.new(assignment)
+    end
+
+    def self.get_broker_metadatas(zk_client, brokers, force_rack = true)
+      rack_aware = if force_rack
+        JRuby.runtime.jruby_class_loader.load_class('kafka.admin.RackAwareMode$Enforced$').get_declared_field('MODULE$').get(nil)
+      else
+        JRuby.runtime.jruby_class_loader.load_class('kafka.admin.RackAwareMode$Safe$').get_declared_field('MODULE$').get(nil)
+      end
+      broker_metadatas = Kafka::Admin::AdminUtils.get_broker_metadatas(
+        zk_client.utils, 
+        rack_aware,
+        Scala::Option[Scala::Collection::JavaConversions.as_scala_iterable(brokers).to_list]
+      )
+      Scala::Collection::JavaConversions.seq_as_java_list(broker_metadatas).to_a
+    end
+  end
+
+  module Protocol
+    java_import 'org.apache.kafka.common.protocol.SecurityProtocol'
+  end
+
+  module Common
+    include_package 'kafka.common'
+  end
+
+  TopicAndPartition = Common::TopicAndPartition
+
+  module Tools
+    include_package 'kafka.tools'
+  end
+end

data/lib/ext/thor.rb

@@ -0,0 +1,14 @@
+# encoding: utf-8
+
+class Thor
+  module Shell
+    class Basic
+
+      protected
+
+      def unix?
+        (RUBY_PLATFORM =~ /java/i && RbConfig::CONFIG['host_os'] =~ /darwin|linux/i) || super
+      end
+    end
+  end
+end

data/lib/ktl.rb

@@ -0,0 +1,51 @@
+# encoding: utf-8
+
+unless defined?($SLF4J_BACKEND)
+  $SLF4J_BACKEND = 'log4j12'
+end
+
+require 'thor'
+require 'json'
+require 'logger'
+require 'ext/kafka'
+require 'ext/thor'
+
+module Ktl
+  KtlError = Class.new(StandardError)
+  InsufficientBrokersRemainingError = Class.new(KtlError)
+
+  CanBuildFrom = Scala::Collection::Immutable::List.can_build_from
+
+  module JavaConcurrent
+    include_package 'java.util.concurrent'
+  end
+
+  class NullLogger
+    def close(*); end
+    def debug(*); end
+    def debug?; false end
+    def error(*); end
+    def error?; false end
+    def fatal(*); end
+    def fatal?; false end
+    def info(*); end
+    def info?; false end
+    def unknown(*); end
+    def warn(*); end
+    def warn?; false end
+  end
+end
+
+require 'ktl/command'
+require 'ktl/cluster'
+require 'ktl/cluster_stats_task'
+require 'ktl/decommission_plan'
+require 'ktl/migration_plan'
+require 'ktl/reassigner'
+require 'ktl/reassignment_progress'
+require 'ktl/reassignment_task'
+require 'ktl/shuffle_plan'
+require 'ktl/shell_formatter'
+require 'ktl/topic'
+require 'ktl/cli'
+require 'ktl/zookeeper_client'

data/lib/ktl/cli.rb

@@ -0,0 +1,11 @@
+# encoding: utf-8
+
+module Ktl
+  class Cli < Thor
+    desc 'cluster SUBCOMMAND ...ARGS', 'Commands for managing a cluster'
+    subcommand 'cluster', Cluster
+
+    desc 'topic SUBCOMMAND ...ARGS', 'Commands for managing topics'
+    subcommand 'topic', Topic
+  end
+end

data/lib/ktl/cluster.rb

@@ -0,0 +1,108 @@
+# encoding: utf-8
+
+module Ktl
+  class Cluster < Command
+    desc 'stats', 'Show statistics about cluster'
+    option :zookeeper, aliases: %w[-z], required: true, desc: 'ZooKeeper URI'
+    def stats
+      with_zk_client do |zk_client|
+        task = ClusterStatsTask.new(zk_client, shell)
+        task.execute
+      end
+    end
+
+    desc 'preferred-replica [REGEXP]', 'Perform preferred replica leader elections'
+    option :zookeeper, aliases: %w[-z], required: true, desc: 'ZooKeeper URI'
+    def preferred_replica(regexp='.*')
+      with_zk_client do |zk_client|
+        regexp = Regexp.new(regexp)
+        partitions = zk_client.all_partitions
+        partitions = partitions.filter { |tp| !!tp.topic.match(regexp) }.to_set
+        if partitions.size > 0
+          logger.info 'performing preferred replica leader election on %d partitions' % partitions.size
+          Kafka::Admin.preferred_replica(zk_client.raw_client, partitions)
+        else
+          logger.info 'no topics matched %s' % regexp.inspect
+        end
+      end
+    end
+
+    desc 'migrate-broker', 'Migrate partitions from one broker to another'
+    option :from, aliases: %w[-f], type: :numeric, required: true, desc: 'Broker ID of old leader'
+    option :to, aliases: %w[-t], type: :numeric, required: true, desc: 'Broker ID of new leader'
+    option :zookeeper, aliases: %w[-z], required: true, desc: 'ZooKeeper URI'
+    option :limit, aliases: %w[-l], type: :numeric, desc: 'Max number of partitions to reassign at a time'
+    def migrate_broker
+      with_zk_client do |zk_client|
+        old_leader, new_leader = options.values_at(:from, :to)
+        plan = MigrationPlan.new(zk_client, old_leader, new_leader)
+        reassigner = Reassigner.new(zk_client, limit: options.limit, logger: logger)
+        execute_reassignment(reassigner, plan)
+      end
+    end
+
+    desc 'shuffle [REGEXP]', 'Shuffle leaders and replicas for partitions'
+    option :brokers, type: :array, desc: 'Broker IDs'
+    option :blacklist, type: :array, desc: 'Broker IDs to exclude'
+    option :rendezvous, aliases: %w[-R], type: :boolean, desc: 'Whether to use Rendezvous-hashing based shuffle'
+    option :rack_aware, aliases: %w[-a], type: :boolean, desc: 'Whether to use Rack aware + Rendezvous-hashing based shuffle'
+    option :replication_factor, aliases: %w[-r], type: :numeric, desc: 'Replication factor to use'
+    option :limit, aliases: %w[-l], type: :numeric, desc: 'Max number of partitions to reassign at a time'
+    option :zookeeper, aliases: %w[-z], required: true, desc: 'ZooKeeper URI'
+    option :verbose, aliases: %w[-v], desc: 'Verbose output'
+    option :dryrun, aliases: %w[-d], desc: 'Output reassignment plan without executing'
+    def shuffle(regexp='.*')
+      with_zk_client do |zk_client|
+        plan_factory = if options.rack_aware
+          RackAwareShufflePlan
+        elsif options.rendezvous
+          RendezvousShufflePlan
+        else
+          ShufflePlan
+        end
+        plan = plan_factory.new(zk_client, {
+          filter: Regexp.new(regexp),
+          brokers: options.brokers,
+          blacklist: options.blacklist,
+          replication_factor: options.replication_factor,
+          logger: logger,
+          log_plan: options.dryrun,
+        })
+        reassigner = Reassigner.new(zk_client, limit: options.limit, logger: logger, log_assignments: options.verbose)
+        execute_reassignment(reassigner, plan, options.dryrun)
+      end
+    end
+
+    desc 'decommission-broker BROKER_ID', 'Decommission a broker'
+    option :limit, aliases: %w[-l], type: :numeric, desc: 'Max number of partitions to reassign at a time'
+    option :rendezvous, aliases: %w[-R], type: :boolean, desc: 'Whether to use Rendezvous-hashing'
+    option :zookeeper, aliases: %w[-z], required: true, desc: 'ZooKeeper URI'
+    def decommission_broker(broker_id)
+      with_zk_client do |zk_client|
+        if options.rendezvous?
+          plan = RendezvousShufflePlan.new(zk_client, blacklist: [broker_id.to_i])
+        else
+          plan = DecommissionPlan.new(zk_client, broker_id.to_i)
+        end
+        reassigner = Reassigner.new(zk_client, limit: options.limit, logger: logger)
+        execute_reassignment(reassigner, plan)
+      end
+    end
+
+    desc 'reassignment-progress', 'Show progress of latest reassignment command'
+    option :verbose, aliases: %w[-v], desc: 'Verbose output'
+    option :zookeeper, aliases: %w[-z], required: true, desc: 'ZooKeeper URI'
+    def reassignment_progress
+      with_zk_client do |zk_client|
+        progress = ReassignmentProgress.new(zk_client, options.merge(logger: logger))
+        progress.display(shell)
+      end
+    end
+
+    private
+
+    def execute_reassignment(reassigner, plan, dryrun = false)
+      ReassignmentTask.new(reassigner, plan, shell, logger: logger).execute(dryrun)
+    end
+  end
+end