clickhouse 0.1.1 → 0.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 382f431988af06b9879728821c0c002676cb930d
-  data.tar.gz: 7fc6c4674ef83c45989bd8d3759ad85629beecee
+  metadata.gz: 99ef193e0967f146e206ce76d41d1b5893948ae9
+  data.tar.gz: 47a7ba893cb391215495c80384146947be07b62d
 SHA512:
-  metadata.gz: 83868f46b3dbbd9e977c6ac14a4badf252f9cc213cfbab9545a777ff241003357bd90241bbea1cedd58871457e0a4c43598385839a827e1815a897848bc5bcab
-  data.tar.gz: 39d8d2ab4a094e5e5f7b3d269d40d6fad75619fd97a3bee5c04b7bd766eff2096398a5c798fad6cf29d6ee0bafbe33d908cd7068e5c0a98e68c37a051d9a79f5
+  metadata.gz: 774e1ae3330a758d3dd5bd0609d4f4819a06050a35f2f66b520fbfced6baab89ab376e5c855079486dccc841e627bfb9db19af48ca5005c08154d9e95974180d
+  data.tar.gz: 06dd73764247a86c0651f50b1a10c27f5b8f8e13e1b9198b5f1386f73edb597c9c51afb8f28f08999a4feaf4f6fc7fefe4ae967af93d29682ead965d34293f67

data/CHANGELOG.md

@@ -1,5 +1,12 @@
 ## Clickhouse CHANGELOG
 
+### Version 0.1.2 (October 20, 2016)
+
+* Being able to specify a URL
+* Raising a Clickhouse::ConnectionError when getting a Faraday::Error
+* Made Clickhouse::Connection::Client#ping! public
+* Supporting cluster connections using Pond (thanks @chanks)
+
 ### Version 0.1.1 (October 19, 2016)
 
 * Using the JSONCompact format as query output which does not brake when having a JSON string within the data

data/README.md

@@ -1,10 +1,10 @@
 # Clickhouse [![Build Status](https://travis-ci.org/archan937/clickhouse.svg?branch=master)](https://travis-ci.org/archan937/clickhouse) [![Code Climate](https://codeclimate.com/github/archan937/clickhouse/badges/gpa.svg)](https://codeclimate.com/github/archan937/clickhouse) [![Test Coverage](https://codeclimate.com/github/archan937/clickhouse/badges/coverage.svg)](https://codeclimate.com/github/archan937/clickhouse/coverage) [![Gem](https://img.shields.io/gem/v/clickhouse.svg)](https://rubygems.org/gems/clickhouse) [![Gem](https://img.shields.io/gem/dt/clickhouse.svg)](https://rubygems.org/gems/clickhouse)
 
-A Ruby database driver for Clickhouse.
+A Ruby database driver for ClickHouse.
 
 ## Introduction
 
-[Clickhouse](https://clickhouse.yandex) is a high-performance column-oriented database management system developed by [Yandex](https://yandex.com/company) which operates Russia's most popular search engine.
+[ClickHouse](https://clickhouse.yandex) is a high-performance column-oriented database management system developed by [Yandex](https://yandex.com/company) which operates Russia's most popular search engine.
 
 > ClickHouse manages extremely large volumes of data in a stable and sustainable manner. It currently powers Yandex.Metrica, world’s second largest web analytics platform, with over 13 trillion database records and over 20 billion events a day, generating customized reports on-the-fly, directly from non-aggregated data. This system was successfully implemented at CERN’s LHCb experiment to store and process metadata on 10bn events with over 1000 attributes per event registered in 2011.
 
@@ -12,14 +12,14 @@ On June 15th 2016, [Yandex open-sourced their awesome project](https://news.ycom
 
 ### Why use the HTTP interface and not the TCP interface?
 
-Well, the developers of Clickhouse themselves [discourage](https://github.com/yandex/ClickHouse/issues/45#issuecomment-231194134) using the TCP interface.
+Well, the developers of ClickHouse themselves [discourage](https://github.com/yandex/ClickHouse/issues/45#issuecomment-231194134) using the TCP interface.
 
 > TCP transport is more specific, we don't want to expose details.
 Despite we have full compatibility of protocol of different versions of client and server, we want to keep the ability to "break" it for very old clients. And that protocol is not too clean to make a specification.
 
 ### Why use the JSONCompact format and not the native format?
 
-Despite of it being the most efficient format, using the native format is also [discouraged] by the Clickhouse developers.
+Despite of it being the most efficient format, using the native format is also [discouraged](https://clickhouse.yandex/reference_en.html#Native) by the ClickHouse developers.
 
 > The most efficient format. Data is written and read by blocks in binary format. For each block, the number of rows, number of columns, column names and types, and parts of columns in this block are recorded one after another. In other words, this format is "columnar" - it doesn't convert columns to rows. This is the format used in the native interface for interaction between servers, for using the command-line client, and for C++ clients.
 >
@@ -48,7 +48,7 @@ require "logger"
 Clickhouse.logger = Logger.new(STDOUT)
 ```
 
-Establish the connection with the Clickhouse server (using the default config).
+Establish the connection with the ClickHouse server (using the default config).
 
 ```ruby
 Clickhouse.establish_connection
@@ -147,7 +147,30 @@ To see what more the `Clickhouse` gem has to offer, please take a look at the un
 
 ## Using the console
 
-As you probably already noticed, the `Clickhouse` repo is provided with a `script/console` file which you can use for development / testing purposes. Please note that you need to have a Clickhouse server running.
+As you probably already noticed, the `Clickhouse` repo is provided with a `script/console` file which you can use for development / testing purposes. Please note that you need to have a ClickHouse server running.
+
+### Running a ClickHouse server on your Mac or Windows computer
+
+Despite that the ClickHouse build is not intended to work on Mac OS X or Windows (only x86_64 with SSE 4.2 is supported), you can still run a ClickHouse server instance on both the operating systems using the [ClickHouse Server Docker Image](https://hub.docker.com/r/yandex/clickhouse-server/) hosted on [https://hub.docker.com/](Docker Hub).
+
+The installation process is just a matter of two simple steps:
+
+* Download and install [Kitematic](https://kitematic.com/) (Docker Toolbox) on your computer
+* Install the [clickhouse-server](https://hub.docker.com/r/yandex/clickhouse-server/) container using Kitematic
+
+Et voilà! Your ClickHouse server instance is up and running locally. Please make sure to use the proper IP address and port to connect with. You can find it at the container details within Kitematic (it is the `Access URL` corresponded with the `8123/tcp Docker port`).
+
+### Example
+
+    $ script/console
+    Loading Clickhouse development environment (0.1.1)
+    [1] pry(main)> connect! host: "192.168.99.100", port: 32770
+    => true
+    [2] pry(main)> conn.databases
+    I, [2016-10-19T20:54:53.081388 #29847]  INFO -- :
+      SQL (3.1ms)  SHOW DATABASES;
+    => ["default", "system"]
+    [3] pry(main)>
 
 ## Testing
 
@@ -163,10 +186,6 @@ You can also run a single test file:
 
 For support, remarks and requests, please mail me at [pm_engel@icloud.com](mailto:pm_engel@icloud.com).
 
-## TODO
-
-* Support cluster connections
-
 ## License
 
 Copyright (c) 2016 Paul Engel, released under the MIT license

data/VERSION

@@ -1 +1 @@
-0.1.1
+0.1.2

data/clickhouse.gemspec

@@ -4,8 +4,8 @@ require File.expand_path("../lib/clickhouse/version", __FILE__)
 Gem::Specification.new do |gem|
   gem.authors       = ["Paul Engel"]
   gem.email         = ["pm_engel@icloud.com"]
-  gem.summary       = %q{A Ruby database driver for Clickhouse}
-  gem.description   = %q{A Ruby database driver for Clickhouse}
+  gem.summary       = %q{A Ruby database driver for ClickHouse}
+  gem.description   = %q{A Ruby database driver for ClickHouse}
   gem.homepage      = "https://github.com/archan937/clickhouse"
 
   gem.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
@@ -16,6 +16,7 @@ Gem::Specification.new do |gem|
   gem.version       = Clickhouse::VERSION
 
   gem.add_dependency "faraday"
+  gem.add_dependency "pond"
 
   gem.add_development_dependency "rake"
   gem.add_development_dependency "pry"

data/lib/clickhouse.rb

@@ -1,8 +1,11 @@
+require "uri"
 require "forwardable"
 require "csv"
 
 require "faraday"
+require "pond"
 
+require "clickhouse/cluster"
 require "clickhouse/connection"
 require "clickhouse/error"
 require "clickhouse/version"
@@ -28,8 +31,7 @@ module Clickhouse
   def self.establish_connection(arg = {})
     config = arg.is_a?(Hash) ? arg : (configurations || {})[arg.to_s]
     if config
-      @connection = Connection.new(config)
-      @connection.connect!
+      connect!(config)
     else
       raise InvalidConnectionError, "Invalid connection specified: #{arg.inspect}"
     end
@@ -39,4 +41,14 @@ module Clickhouse
     @connection if instance_variables.include?(:@connection)
   end
 
+# private
+
+  def self.connect!(config)
+    klass = (config[:urls] || config["urls"]) ? Cluster : Connection
+    @connection = klass.new(config)
+    @connection.connect!
+  end
+
+  private_class_method :connect!
+
 end

data/lib/clickhouse/cluster.rb

@@ -0,0 +1,41 @@
+module Clickhouse
+  class Cluster < BasicObject
+
+    attr_reader :pond
+
+    def initialize(config)
+      urls = config.delete(:urls) || config.delete("urls")
+      @pond = ::Pond.new :maximum_size => urls.size, :timeout => 0.1
+
+      block = ::Proc.new do
+        url = (urls - pond.available.collect(&:url)).first || urls.sample
+        ::Clickhouse::Connection.new(config.merge(:url => url))
+      end
+
+      pond.instance_variable_set :@block, block
+      pond.maximum_size.times do
+        pond.available << block.call
+      end
+
+      pond.detach_if = ::Proc.new do |connection|
+        begin
+          connection.ping!
+          false
+        rescue
+          true
+        end
+      end
+    end
+
+  private
+
+    def method_missing(*args, &block)
+      pond.checkout do |connection|
+        connection.send(*args, &block)
+      end
+    rescue
+      retry if pond.available.any?
+    end
+
+  end
+end

data/lib/clickhouse/connection.rb

@@ -5,18 +5,37 @@ require "clickhouse/connection/query"
 module Clickhouse
   class Connection
 
+    DEFAULT_CONFIG = {
+      :scheme => "http",
+      :host => "localhost",
+      :port => 8123
+    }
+
     include Client
     include Logger
     include Query
 
     def initialize(config = {})
-      @config = {
-        :scheme => "http",
-        :host => "localhost",
-        :port => 8123
-      }.merge(
-        config.inject({}){|h, (k, v)| h[k.to_sym] = v; h}
-      )
+      @config = normalize_config(config)
+    end
+
+  private
+
+    def normalize_config(config)
+      config = config.inject({}) do |hash, (key, value)|
+        hash[key.to_sym] = value
+        hash
+      end
+
+      if config[:url]
+        uri = URI config[:url]
+        config[:scheme] = uri.scheme
+        config[:host] = uri.host
+        config[:port] = uri.port
+        config.delete(:url)
+      end
+
+      DEFAULT_CONFIG.merge(config)
     end
 
   end

data/lib/clickhouse/connection/client.rb

@@ -3,9 +3,18 @@ module Clickhouse
     module Client
 
       def connect!
-        return if connected?
+        ping! unless connected?
+      end
+
+      def ping!
         ensure_authentication
-        ping!
+        status = client.get("/").status
+        if status != 200
+          raise ConnectionError, "Unexpected response status: #{status}"
+        end
+        true
+      rescue Faraday::Error => e
+        raise ConnectionError, e.message
       end
 
       def connected?
@@ -20,12 +29,12 @@ module Clickhouse
         request(:post, query, body)
       end
 
-    private
-
       def url
         "#{@config[:scheme]}://#{@config[:host]}:#{@config[:port]}"
       end
 
+    private
+
       def path(query)
         database = "database=#{@config[:database]}&" if @config[:database]
         "/?#{database}query=#{CGI.escape(query)}"
@@ -40,16 +49,6 @@ module Clickhouse
         client.basic_auth(username || "default", password) if username || password
       end
 
-      def ping!
-        status = client.get("/").status
-        if status != 200
-          raise ConnectionError, "Unexpected response status: #{status}"
-        end
-        true
-      rescue Faraday::ConnectionFailed => e
-        raise ConnectionError, e.message
-      end
-
       def request(method, query, body = nil)
         connect!
         query = query.to_s.strip
@@ -58,6 +57,8 @@ module Clickhouse
           log :info, "\n  SQL (#{((Time.now - start) * 1000).round(1)}ms)  #{query.gsub(/( FORMAT \w+|;$)/, "")};"
           raise QueryError, response.body unless response.status == 200
         end
+      rescue Faraday::Error => e
+        raise ConnectionError, e.message
       end
 
     end

data/lib/clickhouse/version.rb

@@ -1,7 +1,7 @@
 module Clickhouse
   MAJOR = 0
   MINOR = 1
-  TINY  = 1
+  TINY  = 2
 
   VERSION = [MAJOR, MINOR, TINY].join(".")
 end

data/test/unit/connection/test_client.rb

@@ -77,7 +77,7 @@ module Unit
 
         describe "#request" do
           before do
-            @connection.expects(:log)
+            @connection.stubs(:log)
           end
 
           it "connects to the server first" do
@@ -102,6 +102,16 @@ module Unit
               end
             end
           end
+
+          describe "when getting Faraday::Error" do
+            it "raises a Clickhouse::ConnectionError" do
+              @connection.instance_variable_set :@client, (client = mock)
+              client.expects(:get).raises(Faraday::ConnectionFailed.new("Failed to connect"))
+              assert_raises Clickhouse::ConnectionError do
+                @connection.send(:request, :get, "SELECT 1")
+              end
+            end
+          end
         end
 
         describe "configuration" do
@@ -117,8 +127,8 @@ module Unit
 
           describe "authentication" do
             it "includes the credentials in the request headers" do
+              Faraday::Connection.any_instance.expects(:get).returns(stub(status: 200))
               connection = Clickhouse::Connection.new :password => "awesomepassword"
-              connection.expects(:ping!)
               connection.connect!
               assert_equal "Basic ZGVmYXVsdDphd2Vzb21lcGFzc3dvcmQ=", connection.send(:client).headers["Authorization"].force_encoding("UTF-8")
             end

data/test/unit/connection/test_cluster.rb

@@ -0,0 +1,51 @@
+require_relative "../../test_helper"
+
+module Unit
+  module Connection
+    class TestCluser < MiniTest::Test
+
+      describe Clickhouse::Cluster do
+        it "creates a connection pond" do
+          cluster = Clickhouse::Cluster.new :urls => %w(localhost:1234 localhost:1235 localhost:1236)
+          assert_equal true, cluster.pond.is_a?(Pond)
+        end
+
+        describe "when connection succeeds" do
+          it "keeps valid connections from the pond" do
+            Clickhouse::Connection.any_instance.expects(:tables)
+            Clickhouse::Connection.any_instance.expects(:ping!)
+
+            cluster = Clickhouse::Cluster.new :urls => %w(http://localhost:1234 http://localhost:1235 http://localhost:1236)
+            assert_equal %w(
+              http://localhost:1234
+              http://localhost:1235
+              http://localhost:1236
+            ), cluster.pond.available.collect(&:url)
+
+            cluster.tables
+            assert_equal %w(
+              http://localhost:1235
+              http://localhost:1236
+              http://localhost:1234
+            ), cluster.pond.available.collect(&:url)
+          end
+        end
+
+        describe "when connection fails" do
+          it "removes invalid connections from the pond" do
+            cluster = Clickhouse::Cluster.new :urls => %w(http://localhost:1234 http://localhost:1235 http://localhost:1236)
+            assert_equal %w(
+              http://localhost:1234
+              http://localhost:1235
+              http://localhost:1236
+            ), cluster.pond.available.collect(&:url)
+
+            cluster.tables
+            assert_equal [], cluster.pond.available.collect(&:url)
+          end
+        end
+      end
+
+    end
+  end
+end

data/test/unit/test_clickhouse.rb

@@ -62,6 +62,14 @@ module Unit
             Clickhouse::Connection.expects(:new).with(config).returns(@connection)
             Clickhouse.establish_connection "foo"
           end
+
+          describe "cluster connections" do
+            it "creates a connection pool" do
+              config = {:urls => %w(localhost:1234 localhost:1235 localhost:1236)}
+              Clickhouse::Cluster.expects(:new).with(config).returns(@connection)
+              Clickhouse.establish_connection config
+            end
+          end
         end
 
         describe "when invalid" do

data/test/unit/test_connection.rb

@@ -37,6 +37,17 @@ module Unit
             }, connection.instance_variable_get(:@config))
           end
         end
+
+        describe "when passing a URL" do
+          it "derives the scheme, host and port" do
+            connection = Clickhouse::Connection.new :url => "https://19.82.8.1:1947"
+            assert_equal({
+              :scheme => "https",
+              :host => "19.82.8.1",
+              :port => 1947
+            }, connection.instance_variable_get(:@config))
+          end
+        end
       end
     end
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: clickhouse
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.1.2
 platform: ruby
 authors:
 - Paul Engel
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-10-19 00:00:00.000000000 Z
+date: 2016-10-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: faraday
@@ -24,6 +24,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: pond
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
@@ -122,7 +136,7 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-description: A Ruby database driver for Clickhouse
+description: A Ruby database driver for ClickHouse
 email:
 - pm_engel@icloud.com
 executables: []
@@ -139,6 +153,7 @@ files:
 - VERSION
 - clickhouse.gemspec
 - lib/clickhouse.rb
+- lib/clickhouse/cluster.rb
 - lib/clickhouse/connection.rb
 - lib/clickhouse/connection/client.rb
 - lib/clickhouse/connection/logger.rb
@@ -157,6 +172,7 @@ files:
 - test/unit/connection/query/test_result_set.rb
 - test/unit/connection/query/test_table.rb
 - test/unit/connection/test_client.rb
+- test/unit/connection/test_cluster.rb
 - test/unit/connection/test_logger.rb
 - test/unit/connection/test_query.rb
 - test/unit/test_clickhouse.rb
@@ -183,7 +199,7 @@ rubyforge_project:
 rubygems_version: 2.4.5.1
 signing_key: 
 specification_version: 4
-summary: A Ruby database driver for Clickhouse
+summary: A Ruby database driver for ClickHouse
 test_files:
 - test/test_helper.rb
 - test/test_helper/coverage.rb
@@ -193,6 +209,7 @@ test_files:
 - test/unit/connection/query/test_result_set.rb
 - test/unit/connection/query/test_table.rb
 - test/unit/connection/test_client.rb
+- test/unit/connection/test_cluster.rb
 - test/unit/connection/test_logger.rb
 - test/unit/connection/test_query.rb
 - test/unit/test_clickhouse.rb