trino-client 1.0.0

data/README.md

@@ -0,0 +1,131 @@
+# Trino client library for Ruby
+
+[![Build Status](https://travis-ci.org/treasure-data/trino-client-ruby.svg?branch=master)](https://travis-ci.org/treasure-data/trino-client-ruby) [![Gem](https://img.shields.io/gem/v/trino-client)](https://rubygems.org/gems/trino-client) [![Gem](https://img.shields.io/gem/dt/trino-client)](https://rubygems.org/gems/trino-client) [![GitHub](https://img.shields.io/github/license/treasure-data/trino-client-ruby)]()
+
+Trino is a distributed SQL query engine for big data:
+https://trino.io/
+
+This is a client library for Ruby to run queries on Trino.
+
+## Example
+
+```ruby
+require 'trino-client'
+
+# create a client object:
+client = Trino::Client.new(
+  server: "localhost:8880",   # required option
+  ssl: {verify: false},
+  catalog: "native",
+  schema: "default",
+  user: "frsyuki",
+  password: "********",
+  time_zone: "US/Pacific",
+  language: "English",
+  properties: {
+    "hive.force_local_scheduling": true,
+    "raptor.reader_stream_buffer_size": "32MB"
+  },
+  http_proxy: "proxy.example.com:8080",
+  http_debug: true
+)
+
+# run a query and get results as an array of arrays:
+columns, rows = client.run("select * from sys.node")
+rows.each {|row|
+  p row  # row is an array
+}
+
+# run a query and get results as an array of hashes:
+results = client.run_with_names("select alpha, 1 AS beta from tablename")
+results.each {|row|
+  p row['alpha']   # access by name
+  p row['beta']
+  p row.values[0]  # access by index
+  p row.values[1]
+}
+
+# run a query and fetch results streamingly:
+client.query("select * from sys.node") do |q|
+  # get columns:
+  q.columns.each {|column|
+    puts "column: #{column.name}.#{column.type}"
+  }
+
+  # get query results. it feeds more rows until
+  # query execution finishes:
+  q.each_row {|row|
+    p row  # row is an array
+  }
+end
+
+# killing a query
+query = client.query("select * from sys.node")
+query_id = query.query_info.query_id
+query.each_row {|row| ... }  # when a thread is processing the query,
+client.kill(query_id)  # another thread / process can kill the query.
+```
+
+## Build models
+
+```
+$ bundle exec rake modelgen:latest
+```
+
+## Options
+
+* **server** sets address (and port) of a Trino coordinator server.
+* **ssl** enables https.
+  * Setting `true` enables SSL and verifies server certificate using system's built-in certificates.
+  * Setting `{verify: false}` enables SSL but doesn't verify server certificate.
+  * Setting a Hash object enables SSL and verify server certificate with options:
+    * **ca_file**: path of a CA certification file in PEM format
+    * **ca_path**: path of a CA certification directory containing certifications in PEM format
+    * **cert_store**: a `OpenSSL::X509::Store` object used for verification
+    * **client_cert**: a `OpenSSL::X509::Certificate` object as client certificate
+    * **client_key**: a `OpenSSL::PKey::RSA` or `OpenSSL::PKey::DSA` object used for client certificate
+* **catalog** sets catalog (connector) name of Trino such as `hive-cdh4`, `hive-hadoop1`, etc.
+* **schema** sets default schema name of Trino. You need to use qualified name like `FROM myschema.table1` to use non-default schemas.
+* **source** sets source name to connect to a Trino. This name is shown on Trino web interface.
+* **client_info** sets client info to queries. It can be a string to pass a raw string, or an object that can be encoded to JSON.
+* **client_tags** sets client tags to queries. It needs to be an array of strings. The tags are shown on web interface.
+* **user** sets user name to connect to a Trino.
+* **password** sets a password to connect to Trino using basic auth.
+* **time_zone** sets time zone of queries. Time zone affects some functions such as `format_datetime`.
+* **language** sets language of queries. Language affects some functions such as `format_datetime`.
+* **properties** set session properties. Session properties affect internal behavior such as `hive.force_local_scheduling: true`, `raptor.reader_stream_buffer_size: "32MB"`, etc.
+* **query_timeout** sets timeout in seconds for the entire query execution (from the first API call until there're no more output data). If timeout happens, client raises TrinoQueryTimeoutError. Default is nil (disabled).
+* **plan_timeout** sets timeout in seconds for query planning execution (from the first API call until result columns become available). If timeout happens, client raises TrinoQueryTimeoutError. Default is nil (disabled).
+* **http_headers** sets custom HTTP headers. It must be a Hash of string to string.
+* **http_proxy** sets host:port of a HTTP proxy server.
+* **http_debug** enables debug message to STDOUT for each HTTP requests.
+* **http_open_timeout** sets timeout in seconds to open new HTTP connection.
+* **http_timeout** sets timeout in seconds to read data from a server.
+* **gzip** enables gzip compression.
+* **follow_redirect** enables HTTP redirection support.
+* **model_version** set the Trino version to which a job is submitted. Supported versions are 351, 316, 303, 0.205, 0.178, 0.173, 0.153 and 0.149. Default is 351.
+
+See [RDoc](http://www.rubydoc.info/gems/presto-client/) for the full documentation.
+
+## Development
+
+### Releasing a new version
+
+1. First update `lib/trino/client/version.rb` to the next version.
+2. Run the following command which will update `ChangeLog.md` file automatically.
+```
+$ ruby release.rb
+```
+
+3. Create tag
+```
+$ git commit -am "vX.Y.Z"
+$ git tag "vX.Y.Z"
+% git push --tags
+```
+
+4. Push package
+```
+$ gem build trino-client.gemspec
+$ gem push trino-client-X.Y.Z.gem
+```

data/Rakefile

@@ -0,0 +1,45 @@
+#!/usr/bin/env rake
+require 'bundler/gem_tasks'
+
+require 'rake/testtask'
+require 'rake/clean'
+
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => [:spec, :build]
+
+GEN_MODEL_VERSIONS = %w[
+  351
+]
+
+namespace "modelgen" do
+  task :latest => :all do
+    require 'erb'
+    erb = ERB.new(File.read("modelgen/models.rb"))
+    @versions = GEN_MODEL_VERSIONS
+    @latest_version = GEN_MODEL_VERSIONS.last
+    data = erb.result
+    File.write("lib/trino/client/models.rb", data)
+  end
+
+  task :all => GEN_MODEL_VERSIONS
+
+  GEN_MODEL_VERSIONS.each do |ver|
+    file "build/trino-#{ver}.tar.gz" do
+      mkdir_p "build"
+      sh "curl -L -o build/trino-#{ver}.tar.gz https://github.com/trinodb/trino/archive/#{ver}.tar.gz"
+    end
+
+    file "lib/trino/client/model_versions/#{ver}.rb" => "build/trino-#{ver}.tar.gz" do
+      sh "tar zxf build/trino-#{ver}.tar.gz -C build"
+      mkdir_p "lib/trino/client/model_versions"
+      sh "#{RbConfig.ruby} modelgen/modelgen.rb #{ver} build/trino-#{ver} modelgen/model_versions.rb lib/trino/client/model_versions/#{ver}.rb"
+      puts "Generated lib/trino/client/model_versions/#{ver}.rb."
+    end
+
+    task ver => "lib/trino/client/model_versions/#{ver}.rb"
+  end
+end
+

data/lib/trino-client.rb

@@ -0,0 +1 @@
+require 'trino/client'

data/lib/trino/client.rb

@@ -0,0 +1,23 @@
+#
+# Trino client for Ruby
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+#
+module Trino
+  module Client
+
+    require 'trino/client/version'
+    require 'trino/client/client'
+
+  end
+end

data/lib/trino/client/client.rb

@@ -0,0 +1,78 @@
+#
+# Trino client for Ruby
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+#
+module Trino::Client
+
+  require 'trino/client/models'
+  require 'trino/client/query'
+
+  class Client
+    def initialize(options)
+      @options = options
+    end
+
+    def query(query, &block)
+      q = Query.start(query, @options)
+      if block
+        begin
+          yield q
+        ensure
+          q.close
+        end
+      else
+        return q
+      end
+    end
+
+    def resume_query(next_uri)
+      return Query.resume(next_uri, @options)
+    end
+
+    def kill(query_id)
+      return Query.kill(query_id, @options)
+    end
+
+    def run(query)
+      q = Query.start(query, @options)
+      begin
+        columns = q.columns
+        if columns.empty?
+          return [], []
+        end
+        return columns, q.rows
+      ensure
+        q.close
+      end
+    end
+
+    # Accepts the raw response from the Trino Client and returns an
+    # array of hashes where you can access the data in each row using the
+    # output name specified in the query with AS:
+    #   SELECT expression AS output_name
+    def run_with_names(query)
+      columns, rows = run(query)
+
+      column_names = columns.map(&:name)
+
+      rows.map do |row|
+        Hash[column_names.zip(row)]
+      end
+    end
+  end
+
+  def self.new(*args)
+    Client.new(*args)
+  end
+end

data/lib/trino/client/errors.rb

@@ -0,0 +1,46 @@
+#
+# Trino client for Ruby
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+#
+module Trino::Client
+  class TrinoError < StandardError
+  end
+
+  class TrinoHttpError < TrinoError
+    def initialize(status, message)
+      super(message)
+      @status = status
+    end
+
+    attr_reader :status
+  end
+
+  class TrinoClientError < TrinoError
+  end
+
+  class TrinoQueryError < TrinoError
+    def initialize(message, query_id, error_code, error_name, failure_info)
+      super(message)
+      @query_id = query_id
+      @error_code = error_code
+      @error_name = error_name
+      @failure_info = failure_info
+    end
+
+    attr_reader :error_code, :error_name, :failure_info
+  end
+
+  class TrinoQueryTimeoutError < TrinoError
+  end
+end

data/lib/trino/client/faraday_client.rb

@@ -0,0 +1,242 @@
+#
+# Trino client for Ruby
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+#
+module Trino::Client
+
+  require 'cgi'
+
+  module TrinoHeaders
+    TRINO_USER = "X-Trino-User"
+    TRINO_SOURCE = "X-Trino-Source"
+    TRINO_CATALOG = "X-Trino-Catalog"
+    TRINO_SCHEMA = "X-Trino-Schema"
+    TRINO_TIME_ZONE = "X-Trino-Time-Zone"
+    TRINO_LANGUAGE = "X-Trino-Language"
+    TRINO_SESSION = "X-Trino-Session"
+    TRINO_CLIENT_INFO = "X-Trino-Client-Info";
+    TRINO_CLIENT_TAGS = "X-Trino-Client-Tags";
+
+    TRINO_CURRENT_STATE = "X-Trino-Current-State"
+    TRINO_MAX_WAIT = "X-Trino-Max-Wait"
+    TRINO_MAX_SIZE = "X-Trino-Max-Size"
+    TRINO_PAGE_SEQUENCE_ID = "X-Trino-Page-Sequence-Id"
+  end
+
+  module PrestoHeaders
+    PRESTO_USER = "X-Presto-User"
+    PRESTO_SOURCE = "X-Presto-Source"
+    PRESTO_CATALOG = "X-Presto-Catalog"
+    PRESTO_SCHEMA = "X-Presto-Schema"
+    PRESTO_TIME_ZONE = "X-Presto-Time-Zone"
+    PRESTO_LANGUAGE = "X-Presto-Language"
+    PRESTO_SESSION = "X-Presto-Session"
+    PRESTO_CLIENT_INFO = "X-Presto-Client-Info";
+    PRESTO_CLIENT_TAGS = "X-Presto-Client-Tags";
+
+    PRESTO_CURRENT_STATE = "X-Presto-Current-State"
+    PRESTO_MAX_WAIT = "X-Presto-Max-Wait"
+    PRESTO_MAX_SIZE = "X-Presto-Max-Size"
+    PRESTO_PAGE_SEQUENCE_ID = "X-Presto-Page-Sequence-Id"
+  end
+
+  HEADERS = {
+    "User-Agent" => "trino-ruby/#{VERSION}",
+  }
+
+  def self.faraday_client(options)
+    server = options[:server]
+    unless server
+      raise ArgumentError, ":server option is required"
+    end
+
+    ssl = faraday_ssl_options(options)
+
+    if options[:password] && !ssl
+      raise ArgumentError, "Protocol must be https when passing a password"
+    end
+
+    url = "#{ssl ? "https" : "http"}://#{server}"
+    proxy = options[:http_proxy] || options[:proxy]  # :proxy is obsoleted
+
+    faraday_options = {url: url, proxy: "#{proxy}"}
+    faraday_options[:ssl] = ssl if ssl
+
+    faraday = Faraday.new(faraday_options) do |faraday|
+      if options[:user] && options[:password]
+        faraday.basic_auth(options[:user], options[:password])
+      end
+      if options[:follow_redirect]
+        faraday.use FaradayMiddleware::FollowRedirects
+      end
+      if options[:gzip]
+        faraday.use FaradayMiddleware::Gzip
+      end
+      faraday.response :logger if options[:http_debug]
+      faraday.adapter Faraday.default_adapter
+    end
+
+    faraday.headers.merge!(HEADERS)
+    faraday.headers.merge!(optional_headers(options))
+
+    return faraday
+  end
+
+  def self.faraday_ssl_options(options)
+    ssl = options[:ssl]
+
+    case ssl
+    when true
+      ssl = {verify: true}
+
+    when Hash
+      verify = ssl.fetch(:verify, true)
+      case verify
+      when true
+        # detailed SSL options. pass through to faraday
+      when nil, false
+        ssl = {verify: false}
+      else
+        raise ArgumentError, "Can't convert #{verify.class} of :verify option of :ssl option to true or false"
+      end
+
+    when nil, false
+      ssl = false
+
+    else
+      raise ArgumentError, "Can't convert #{ssl.class} of :ssl option to true, false, or Hash"
+    end
+
+    return ssl
+  end
+
+  def self.optional_headers(options)
+    usePrestoHeader = false
+    if v = options[:model_version] && v < 351
+      usePrestoHeader = true
+    end
+
+    headers = {}
+    if v = options[:user]
+      if usePrestoHeader
+        headers[PrestoHeaders::PRESTO_USER] = v
+      else
+        headers[TrinoHeaders::TRINO_USER] = v
+      end
+    end
+    if v = options[:source]
+      if usePrestoHeader
+        headers[PrestoHeaders::PRESTO_SOURCE] = v
+      else
+        headers[TrinoHeaders::TRINO_SOURCE] = v
+      end
+    end
+    if v = options[:catalog]
+      if usePrestoHeader
+        headers[PrestoHeaders::PRESTO_CATALOG] = v
+      else
+        headers[TrinoHeaders::TRINO_CATALOG] = v
+      end
+    end
+    if v = options[:schema]
+      if usePrestoHeader
+        headers[PrestoHeaders::PRESTO_SCHEMA] = v
+      else
+        headers[TrinoHeaders::TRINO_SCHEMA] = v
+      end
+    end
+    if v = options[:time_zone]
+      if usePrestoHeader
+        headers[PrestoHeaders::PRESTO_TIME_ZONE] = v
+      else
+        headers[TrinoHeaders::TRINO_TIME_ZONE] = v
+      end
+    end
+    if v = options[:language]
+      if usePrestoHeader
+        headers[PrestoHeaders::PRESTO_LANGUAGE] = v
+      else
+        headers[TrinoHeaders::TRINO_LANGUAGE] = v
+      end
+    end
+    if v = options[:properties]
+      if usePrestoHeader
+        headers[PrestoHeaders::PRESTO_SESSION] = encode_properties(v)
+      else
+        headers[TrinoHeaders::TRINO_SESSION] = encode_properties(v)
+      end
+    end
+    if v = options[:client_info]
+      if usePrestoHeader
+        headers[PrestoHeaders::PRESTO_CLIENT_INFO] = encode_client_info(v)
+      else
+        headers[TrinoHeaders::TRINO_CLIENT_INFO] = encode_client_info(v)
+      end
+    end
+    if v = options[:client_tags]
+      if usePrestoHeader
+        headers[PrestoHeaders::PRESTO_CLIENT_TAGS] = encode_client_tags(v)
+      else
+        headers[TrinoHeaders::TRINO_CLIENT_TAGS] = encode_client_tags(v)
+      end
+    end
+    if options[:enable_x_msgpack]
+      # option name is enable_"x"_msgpack because "Accept: application/x-msgpack" header is
+      # not officially supported by Trino. We can use this option only if a proxy server
+      # decodes & encodes response body. Once this option is supported by Trino, option
+      # name should be enable_msgpack, which might be slightly different behavior.
+      headers['Accept'] = 'application/x-msgpack,application/json'
+    end
+    if v = options[:http_headers]
+      headers.merge!(v)
+    end
+    headers
+  end
+
+  HTTP11_SEPARATOR = ["(", ")", "<", ">", "@", ",", ";", ":", "\\", "<", ">", "/", "[", "]", "?", "=", "{", "}", " ", "\v"]
+  HTTP11_TOKEN_CHARSET = (32..126).map {|x| x.chr } - HTTP11_SEPARATOR
+  HTTP11_TOKEN_REGEXP = /^[#{Regexp.escape(HTTP11_TOKEN_CHARSET.join)}]+\z/
+  HTTP11_CTL_CHARSET = (0..31).map {|x| x.chr } + [127.chr]
+  HTTP11_CTL_CHARSET_REGEXP = /[#{Regexp.escape(HTTP11_CTL_CHARSET.join)}]/
+
+  def self.encode_properties(properties)
+    properties.map do |k, v|
+      token = k.to_s
+      field_value = v.to_s  # TODO LWS encoding is not implemented
+      unless k =~ HTTP11_TOKEN_REGEXP
+        raise Faraday::ClientError, "Key of properties can't include HTTP/1.1 control characters or separators (#{HTTP11_SEPARATOR.map {|c| c =~ /\s/ ? c.dump : c }.join(' ')})"
+      end
+      if field_value =~ HTTP11_CTL_CHARSET_REGEXP
+        raise Faraday::ClientError, "Value of properties can't include HTTP/1.1 control characters"
+      end
+      field_value = CGI.escape(field_value)
+      "#{token}=#{field_value}"
+    end
+  end
+
+  def self.encode_client_info(info)
+    if info.is_a?(String)
+      info
+    else
+      JSON.dump(info)
+    end
+  end
+
+  def self.encode_client_tags(tags)
+    Array(tags).join(",")
+  end
+
+  private_class_method :faraday_ssl_options, :optional_headers, :encode_properties, :encode_client_info, :encode_client_tags
+
+end