clickhouse 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: c0be7eb44c6eed3df458e031647a9cad1bc40181
+  data.tar.gz: 724c2dbce96c466eaad983d720f7da01a899571d
+SHA512:
+  metadata.gz: 2c2e727bbcffefa01ae315b19cddbddf6050afefaf82f8b5379496d79ba7add71e5aa0fed8b977c2d8fd012a12affbf7179654cd3fe61fac99d49222fb9820b7
+  data.tar.gz: 5893a3e5919b5d19779f4ffd35f3858a10b33e6e2b7eb9510501d9c7e528be01c256890c53d92d329856e0b85fcb36d9a7398a21840c2472df1939a1fb9345d2

data/.gitignore

@@ -0,0 +1,9 @@
+.DS_Store
+.bundle
+.env
+.rvmrc
+Gemfile.lock
+coverage
+doc
+pkg
+test/coverage

data/.travis.yml

@@ -0,0 +1,5 @@
+language: ruby
+rvm:
+  - 2.3.1
+  - 2.1.2
+  - 2.0.0

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+## Clickhouse CHANGELOG
+
+### Version 0.1.0 (October 18, 2016)
+
+* Initial release

data/Gemfile

@@ -0,0 +1,3 @@
+source "https://rubygems.org"
+
+gemspec

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2016 Paul Engel
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,172 @@
+# Clickhouse [![Build Status](https://travis-ci.org/archan937/clickhouse.svg?branch=master)](https://travis-ci.org/archan937/clickhouse) [![Code Climate](https://codeclimate.com/github/archan937/clickhouse/badges/gpa.svg)](https://codeclimate.com/github/archan937/clickhouse) [![Test Coverage](https://codeclimate.com/github/archan937/clickhouse/badges/coverage.svg)](https://codeclimate.com/github/archan937/clickhouse/coverage) [![Gem](https://img.shields.io/gem/v/clickhouse.svg)](https://rubygems.org/gems/clickhouse) [![Gem](https://img.shields.io/gem/dt/clickhouse.svg)](https://rubygems.org/gems/clickhouse)
+
+A Ruby database driver for Clickhouse.
+
+## Introduction
+
+[Clickhouse](https://clickhouse.yandex) is a high-performance column-oriented database management system developed by [Yandex](https://yandex.com/company) which operates Russia's most popular search engine.
+
+> ClickHouse manages extremely large volumes of data in a stable and sustainable manner. It currently powers Yandex.Metrica, world’s second largest web analytics platform, with over 13 trillion database records and over 20 billion events a day, generating customized reports on-the-fly, directly from non-aggregated data. This system was successfully implemented at CERN’s LHCb experiment to store and process metadata on 10bn events with over 1000 attributes per event registered in 2011.
+
+On June 15th 2016, [Yandex open-sourced their awesome project](https://news.ycombinator.com/item?id=11908254) giving the community a [powerful asset](https://clickhouse.yandex/benchmark.html) which can compete with the big players like [Google BigQuery](https://cloud.google.com/bigquery/) and [Amazon Redshift](http://docs.aws.amazon.com/redshift/latest/mgmt/welcome.html) with an important advantage: the client can use ClickHouse in its infrastructure and does not have to pay for the cloud ([read more](https://translate.google.com/translate?sl=ru&tl=en&js=y&prev=_t&hl=en&ie=UTF-8&u=https://habrahabr.ru/company/yandex/blog/303282/)).
+
+### Why use the HTTP interface and not the TCP interface?
+
+Well, the developers of Clickhouse themselves [discourage](https://github.com/yandex/ClickHouse/issues/45#issuecomment-231194134) using the TCP interface.
+
+> TCP transport is more specific, we don't want to expose details.
+Despite we have full compatibility of protocol of different versions of client and server, we want to keep the ability to "break" it for very old clients. And that protocol is not too clean to make a specification.
+
+## Installation
+
+Run the following command to install `Clickhouse`:
+
+    $ gem install "clickhouse"
+
+## Usage
+
+### Quick start
+
+Require the Clickhouse gem.
+
+```ruby
+require "clickhouse"
+```
+
+Setup the logging output.
+
+```ruby
+require "logger"
+Clickhouse.logger = Logger.new(STDOUT)
+```
+
+Establish the connection with the Clickhouse server (using the default config).
+
+```ruby
+Clickhouse.establish_connection
+=> true
+```
+
+List databases and tables.
+
+```ruby
+Clickhouse.connection.databases
+I, [2016-10-17T22:54:26.587401 #81829]  INFO -- :
+  SQL (64.0ms)  SHOW DATABASES;
+=> ["default", "system"]
+
+Clickhouse.connection.tables
+I, [2016-10-17T22:54:51.454012 #81829]  INFO -- :
+  SQL (61.7ms)  SHOW TABLES;
+=> []
+```
+
+Create tables.
+
+```ruby
+Clickhouse.connection.create_table("events") do |t|
+  t.fixed_string :id, 16
+  t.uint16       :year
+  t.date         :date
+  t.date_time    :time
+  t.string       :event
+  t.uint32       :user_id
+  t.float32      :revenue
+  t.engine       "MergeTree(date, (year, date), 8192)"
+end
+=> true
+
+Clickhouse.connection.query "DESCRIBE TABLE events" # or Clickhouse.connection.describe_table "events"
+=> #<Clickhouse::Connection::Query::ResultSet:0x007fa9ac137010
+ @names=["name", "type", "default_type", "default_expression"],
+ @rows=
+  [["id", "FixedString(16)", nil, nil],
+   ["year", "UInt16", nil, nil],
+   ["date", "Date", nil, nil],
+   ["time", "DateTime", nil, nil],
+   ["event", "String", nil, nil],
+   ["user_id", "UInt32", nil, nil],
+   ["revenue", "Float32", nil, nil]],
+ @types=["String", "String", "String", "String"]>
+```
+
+Insert data.
+
+```ruby
+Clickhouse.connection.insert_rows(events, :names => %w(id year date time event user_id revenue)) do |rows|
+  rows << [
+    "d91d1c90",
+    2016,
+    "2016-10-17",
+    "2016-10-17 23:14:28",
+    "click",
+    1982,
+    0.18
+  ]
+  rows << [
+    "d91d2294",
+    2016,
+    "2016-10-17",
+    "2016-10-17 23:14:41",
+    "click",
+    1947,
+    0.203
+  ]
+end
+=> true
+```
+
+Query data.
+
+```ruby
+Clickhouse.connection.count :from => "events"
+I, [2016-10-17T23:19:45.592602 #82196]  INFO -- :
+  SQL (65.4ms)  SELECT COUNT(*)
+FROM events;
+=> 2
+
+Clickhouse.connection.select_row :select => "COUNT(*), year, date, avg(revenue)", :from => "events", :group => "year, date"
+I, [2016-10-17T23:22:47.340232 #82196]  INFO -- :
+  SQL (67.7ms)  SELECT COUNT(*), year, date, avg(revenue)
+FROM events
+GROUP BY year, date;
+=> [2, 2016, #<Date: 2016-10-17 ((2457679j,0s,0n),+0s,2299161j)>, 0.1915000081062317]
+```
+
+### Check out the tests
+
+To see what more the `Clickhouse` gem has to offer, please take a look at the unit tests ( [test/unit/connection/test_query.rb](https://github.com/archan937/clickhouse/blob/master/test/unit/connection/test_query.rb) for instance).
+
+## Using the console
+
+As you probably already noticed, the `Clickhouse` repo is provided with a `script/console` file which you can use for development / testing purposes. Please note that you need to have a Clickhouse server running.
+
+## Testing
+
+Run the following command for testing:
+
+    $ rake
+
+You can also run a single test file:
+
+    $ ruby test/unit/connection/test_query.rb
+
+## Contact me
+
+For support, remarks and requests, please mail me at [pm_engel@icloud.com](mailto:pm_engel@icloud.com).
+
+## TODO
+
+* Support cluster connections
+
+## License
+
+Copyright (c) 2016 Paul Engel, released under the MIT license
+
+http://github.com/archan937 – http://twitter.com/archan937 – pm_engel@icloud.com
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -0,0 +1,15 @@
+#!/usr/bin/env rake
+
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+task :default => :test
+
+desc "Run tests and report test coverage to Code Climate"
+task :report do
+  exec "REPORT=1 rake"
+end
+
+Rake::TestTask.new do |test|
+  test.pattern = "test/**/test_*.rb"
+end

data/VERSION

@@ -0,0 +1 @@
+0.1.0

data/clickhouse.gemspec

@@ -0,0 +1,27 @@
+# -*- encoding: utf-8 -*-
+require File.expand_path("../lib/clickhouse/version", __FILE__)
+
+Gem::Specification.new do |gem|
+  gem.authors       = ["Paul Engel"]
+  gem.email         = ["pm_engel@icloud.com"]
+  gem.summary       = %q{A Ruby database driver for Clickhouse}
+  gem.description   = %q{A Ruby database driver for Clickhouse}
+  gem.homepage      = "https://github.com/archan937/clickhouse"
+
+  gem.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  gem.files         = `git ls-files`.split("\n")
+  gem.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  gem.name          = "clickhouse"
+  gem.require_paths = ["lib"]
+  gem.version       = Clickhouse::VERSION
+
+  gem.add_dependency "faraday"
+
+  gem.add_development_dependency "rake"
+  gem.add_development_dependency "pry"
+  gem.add_development_dependency "dotenv"
+  gem.add_development_dependency "codeclimate-test-reporter"
+  gem.add_development_dependency "simplecov"
+  gem.add_development_dependency "minitest"
+  gem.add_development_dependency "mocha"
+end

data/lib/clickhouse.rb

@@ -0,0 +1,42 @@
+require "forwardable"
+require "csv"
+
+require "faraday"
+
+require "clickhouse/connection"
+require "clickhouse/error"
+require "clickhouse/version"
+
+module Clickhouse
+
+  def self.logger=(logger)
+    @logger = logger
+  end
+
+  def self.logger
+    @logger if instance_variables.include?(:@logger)
+  end
+
+  def self.configurations=(configurations)
+    @configurations = configurations.inject({}){|h, (k, v)| h[k.to_s] = v; h}
+  end
+
+  def self.configurations
+    @configurations if instance_variables.include?(:@configurations)
+  end
+
+  def self.establish_connection(arg = {})
+    config = arg.is_a?(Hash) ? arg : (configurations || {})[arg.to_s]
+    if config
+      @connection = Connection.new(config)
+      @connection.connect!
+    else
+      raise InvalidConnectionError, "Invalid connection specified: #{arg.inspect}"
+    end
+  end
+
+  def self.connection
+    @connection if instance_variables.include?(:@connection)
+  end
+
+end

data/lib/clickhouse/connection.rb

@@ -0,0 +1,23 @@
+require "clickhouse/connection/client"
+require "clickhouse/connection/logger"
+require "clickhouse/connection/query"
+
+module Clickhouse
+  class Connection
+
+    include Client
+    include Logger
+    include Query
+
+    def initialize(config = {})
+      @config = {
+        :scheme => "http",
+        :host => "localhost",
+        :port => 8123
+      }.merge(
+        config.inject({}){|h, (k, v)| h[k.to_sym] = v; h}
+      )
+    end
+
+  end
+end

data/lib/clickhouse/connection/client.rb

@@ -0,0 +1,65 @@
+module Clickhouse
+  class Connection
+    module Client
+
+      def connect!
+        return if connected?
+        ensure_authentication
+        ping!
+      end
+
+      def connected?
+        instance_variables.include?(:@client) && !!@client
+      end
+
+      def get(query)
+        request(:get, query)
+      end
+
+      def post(query, body = nil)
+        request(:post, query, body)
+      end
+
+    private
+
+      def url
+        "#{@config[:scheme]}://#{@config[:host]}:#{@config[:port]}"
+      end
+
+      def path(query)
+        database = "database=#{@config[:database]}&" if @config[:database]
+        "/?#{database}query=#{CGI.escape(query)}"
+      end
+
+      def client
+        @client ||= Faraday.new(:url => url)
+      end
+
+      def ensure_authentication
+        username, password = @config.values_at(:username, :password)
+        client.basic_auth(username || "default", password) if username || password
+      end
+
+      def ping!
+        status = client.get("/").status
+        if status != 200
+          raise ConnectionError, "Unexpected response status: #{status}"
+        end
+        true
+      rescue Faraday::ConnectionFailed => e
+        raise ConnectionError, e.message
+      end
+
+      def request(method, query, body = nil)
+        connect!
+        query = query.to_s.strip
+        start = Time.now
+        client.send(method, path(query), body).tap do |response|
+          log :info, "\n  SQL (#{((Time.now - start) * 1000).round(1)}ms)  #{query.gsub(/( FORMAT \w+|;$)/, "")};"
+          raise QueryError, response.body unless response.status == 200
+        end
+      end
+
+    end
+  end
+end

data/lib/clickhouse/connection/logger.rb

@@ -0,0 +1,12 @@
+module Clickhouse
+  class Connection
+    module Logger
+    private
+
+      def log(type, msg)
+        Clickhouse.logger.send(type, msg) if Clickhouse.logger
+      end
+
+    end
+  end
+end

data/lib/clickhouse/connection/query.rb

@@ -0,0 +1,160 @@
+require "clickhouse/connection/query/table"
+require "clickhouse/connection/query/result_set"
+require "clickhouse/connection/query/result_row"
+
+module Clickhouse
+  class Connection
+    module Query
+
+      def execute(query, body = nil)
+        body = post(query, body).body.to_s
+        body.empty? ? true : body
+      end
+
+      def query(query)
+        query = query.to_s.gsub(/(;|\bFORMAT \w+)/i, "").strip
+        query += " FORMAT TabSeparatedWithNamesAndTypes"
+        parse_response get(query).body.to_s
+      end
+
+      def databases
+        query("SHOW DATABASES").flatten
+      end
+
+      def tables
+        query("SHOW TABLES").flatten
+      end
+
+      def create_table(name, &block)
+        execute(Clickhouse::Connection::Query::Table.new(name, &block).to_sql)
+      end
+
+      def describe_table(name)
+        query("DESCRIBE TABLE #{name}").to_a
+      end
+
+      def rename_table(*args)
+        names = (args[0].is_a?(Hash) ? args[0].to_a : [args]).flatten
+        raise Clickhouse::InvalidQueryError, "Odd number of table names" unless (names.size % 2) == 0
+        names = Hash[*names].collect{|(from, to)| "#{from} TO #{to}"}
+        execute "RENAME TABLE #{names.join(", ")}"
+      end
+
+      def drop_table(name)
+        execute "DROP TABLE #{name}"
+      end
+
+      def insert_rows(table, options = {})
+        options[:csv] ||= begin
+          options[:rows] ||= yield([])
+          generate_csv options[:rows], options[:names]
+        end
+        execute "INSERT INTO #{table} FORMAT CSVWithNames", options[:csv]
+      end
+
+      def select_rows(options)
+        query to_select_query(options)
+      end
+
+      def select_row(options)
+        select_rows(options)[0]
+      end
+
+      def select_values(options)
+        select_rows(options).collect{|row| row[0]}
+      end
+
+      def select_value(options)
+        values = select_values(options)
+        values[0] if values
+      end
+
+      def count(options)
+        select_value options.merge(:select => "COUNT(*)")
+      end
+
+    private
+
+      def generate_csv(rows, names = nil)
+        hashes = rows[0].is_a?(Hash)
+
+        if hashes
+          names ||= rows[0].keys
+        end
+
+        CSV.generate do |csv|
+          csv << names if names
+          rows.each do |row|
+            csv << (hashes ? row.values_at(*names) : row)
+          end
+        end
+      end
+
+      def inspect_value(value)
+        value.nil? ? "NULL" : value.inspect.gsub(/(^"|"$)/, "'").gsub("\\\"", "\"")
+      end
+
+      def to_select_options(options)
+        keys = [:select, :from, :where, :group, :having, :order, :limit, :offset]
+
+        options = Hash[keys.zip(options.values_at(*keys))]
+        options[:select] ||= "*"
+        options[:limit] ||= 0 if options[:offset]
+        options[:limit] = options.values_at(:offset, :limit).compact.join(", ") if options[:limit]
+        options.delete(:offset)
+
+        options
+      end
+
+      def to_segment(type, value)
+        case type
+        when :select
+          [value].flatten.join(", ")
+        when :where, :having
+          to_condition_statements(value)
+        else
+          value
+        end
+      end
+
+      def to_condition_statements(value)
+        value.collect do |attr, val|
+          if val == :empty
+            "empty(#{attr})"
+          elsif val.is_a?(Range)
+            [
+              "#{attr} >= #{inspect_value(val.first)}",
+              "#{attr} <= #{inspect_value(val.last)}"
+            ]
+          elsif val.is_a?(Array)
+            "#{attr} IN (#{val.collect{|x| inspect_value(x)}.join(", ")})"
+          elsif val.to_s.match(/^`.*`$/)
+            "#{attr} #{val.gsub(/(^`|`$)/, "")}"
+          else
+            "#{attr} = #{inspect_value(val)}"
+          end
+        end.flatten.join(" AND ")
+      end
+
+      def to_select_query(options)
+        to_select_options(options).collect do |(key, value)|
+          next if value.nil? && (!value.respond_to?(:empty?) || value.empty?)
+
+          statement = [key.to_s.upcase]
+          statement << "BY" if %W(GROUP ORDER).include?(statement[0])
+          statement << to_segment(key, value)
+          statement.join(" ")
+
+        end.compact.join("\n").force_encoding("UTF-8")
+      end
+
+      def parse_response(response)
+        rows = CSV.parse response, :col_sep => "\t"
+        names = rows.shift
+        types = rows.shift
+        ResultSet.new rows, names, types
+      end
+
+    end
+  end
+end