rb_snowflake_client 0.1.1 → 0.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/Gemfile.lock +7 -1
- data/README.md +61 -32
- data/lib/ruby_snowflake/client/http_connection_wrapper.rb +2 -0
- data/lib/ruby_snowflake/client/key_pair_jwt_auth_manager.rb +59 -0
- data/lib/ruby_snowflake/client.rb +51 -72
- data/lib/ruby_snowflake/row.rb +10 -5
- data/lib/ruby_snowflake/version.rb +1 -1
- data/rb_snowflake_client.gemspec +6 -0
- metadata +88 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 2a93f0b1e503e2d219db8b1d1133395ca484b45aaa1f91ceef013a50a116b973
|
4
|
+
data.tar.gz: c292c5b0d25a921f8deba6882fe288ed084aa07919fa0d056899d20b921612fc
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 28428d4cf4bd70a11fefad327a2ab14deff17e95bf28b999a6b84d67756b35d3207c8224d6a028b9998ca85b62a7226f4df2beddea50f30e9a91350d7df73b4b
|
7
|
+
data.tar.gz: ad55068885a29f1cae0181e52d66764651cb4f7d541e46680bd15644ed8c17f10dda8cb9356ae4948cdf9b249e0c726abc540e26b03e1b76f9f2c538a8251cce
|
data/Gemfile.lock
CHANGED
data/README.md
CHANGED
@@ -13,6 +13,8 @@ This library is implemented in ruby and while it leverages some libraries that h
|
|
13
13
|
|
14
14
|
# Usage
|
15
15
|
|
16
|
+
## Create a client
|
17
|
+
|
16
18
|
Add to your Gemfile or use `gem install rb-snowflake-client`
|
17
19
|
```ruby
|
18
20
|
gem "rb-snowflake-client"
|
@@ -27,11 +29,12 @@ require "rb_snowflake_client"
|
|
27
29
|
# see: https://github.com/rinsed-org/pure-ruby-snowflake-client/blob/master/lib/ruby_snowflake/client.rb#L43
|
28
30
|
client = RubySnowflake::Client.new(
|
29
31
|
"https://yourinstance.region.snowflakecomputing.com", # insert your URL here
|
30
|
-
File.read("secrets/my_key.pem"), #
|
32
|
+
File.read("secrets/my_key.pem"), # your private key in PEM format (scroll down for instructions)
|
31
33
|
"snowflake-organization", # your account name (doesn't match your URL)
|
32
34
|
"snowflake-account", # typically your subdomain
|
33
35
|
"snowflake-user", # Your snowflake user
|
34
36
|
"some_warehouse", # The name of your warehouse to use by default
|
37
|
+
"some_database", # The name of the database in the context of which the queries will run
|
35
38
|
max_connections: 12, # Config options can be passed in
|
36
39
|
connection_timeout: 45, # See below for the full set of options
|
37
40
|
)
|
@@ -40,20 +43,22 @@ client = RubySnowflake::Client.new(
|
|
40
43
|
RubySnowflake::Client.from_env
|
41
44
|
```
|
42
45
|
Available ENV variables (see below in the config section for details)
|
43
|
-
`SNOWFLAKE_URI`
|
44
|
-
`SNOWFLAKE_PRIVATE_KEY_PATH`
|
45
|
-
|
46
|
-
`
|
47
|
-
`
|
48
|
-
`
|
49
|
-
`
|
50
|
-
`
|
51
|
-
`SNOWFLAKE_JWT_TOKEN_TTL`
|
52
|
-
`SNOWFLAKE_CONNECTION_TIMEOUT`
|
53
|
-
`SNOWFLAKE_MAX_CONNECTIONS`
|
54
|
-
`SNOWFLAKE_MAX_THREADS_PER_QUERY`
|
55
|
-
`SNOWFLAKE_THREAD_SCALE_FACTOR`
|
56
|
-
`SNOWFLAKE_HTTP_RETRIES`
|
46
|
+
- `SNOWFLAKE_URI`
|
47
|
+
- `SNOWFLAKE_PRIVATE_KEY_PATH` or `SNOWFLAKE_PRIVATE_KEY`
|
48
|
+
- Use either the key or the path. Key takes precedence if both are provided.
|
49
|
+
- `SNOWFLAKE_ORGANIZATION`
|
50
|
+
- `SNOWFLAKE_ACCOUNT`
|
51
|
+
- `SNOWFLAKE_USER`
|
52
|
+
- `SNOWFLAKE_DEFAULT_WAREHOUSE`
|
53
|
+
- `SNOWFLAKE_DEFAULT_DATABASE`
|
54
|
+
- `SNOWFLAKE_JWT_TOKEN_TTL`
|
55
|
+
- `SNOWFLAKE_CONNECTION_TIMEOUT`
|
56
|
+
- `SNOWFLAKE_MAX_CONNECTIONS`
|
57
|
+
- `SNOWFLAKE_MAX_THREADS_PER_QUERY`
|
58
|
+
- `SNOWFLAKE_THREAD_SCALE_FACTOR`
|
59
|
+
- `SNOWFLAKE_HTTP_RETRIES`
|
60
|
+
|
61
|
+
## Make queries
|
57
62
|
|
58
63
|
Once you have a client, make queries
|
59
64
|
```ruby
|
@@ -66,37 +71,60 @@ result.each do |row|
|
|
66
71
|
puts row["name"] # or case insensitive strings
|
67
72
|
puts row.to_h # and can produce a hash with keys/values
|
68
73
|
end
|
74
|
+
```
|
75
|
+
|
76
|
+
## Stream results
|
77
|
+
|
78
|
+
You can also stream results and not hold them all in memory. The client will prefetch the next data partition only. If you have some IO in your processing there should usually be data available for you.
|
69
79
|
|
70
|
-
|
71
|
-
# The client will prefetch the next data partition only. If you
|
72
|
-
# have some IO in your processing there should usually be data
|
73
|
-
# available for you.
|
80
|
+
```ruby
|
74
81
|
result = client.query("SELECT * FROM HUGETABLE", streaming: true)
|
75
82
|
result.each do |row|
|
76
83
|
puts row
|
77
84
|
end
|
78
85
|
```
|
79
86
|
|
87
|
+
## Switching databases
|
88
|
+
|
89
|
+
You can also overwrite the database specified in the initializer, and run your query with a different context.
|
90
|
+
|
91
|
+
```ruby
|
92
|
+
result = client.query("SELECT * FROM SECRET_TABLE", database: "OTHER_DB")
|
93
|
+
result.each do |row|
|
94
|
+
puts row
|
95
|
+
end
|
96
|
+
```
|
97
|
+
|
98
|
+
## Switching warehouses
|
99
|
+
|
100
|
+
Clients are not warehouse specific, you can override the default warehouse per query
|
101
|
+
|
102
|
+
```ruby
|
103
|
+
client.query("SELECT * FROM BIGTABLE", warehouse: "FAST_WH")
|
104
|
+
```
|
105
|
+
|
80
106
|
# Configuration Options
|
81
107
|
|
82
|
-
The client supports the following configuration options, each with their own getter/setter except connection pool options which must be set at construction. Additionally, all except logger can be configured with environment variables (see above, but the pattern is like: "SNOWFLAKE_HTTP_RETRIES".
|
108
|
+
The client supports the following configuration options, each with their own getter/setter except connection pool options which must be set at construction. Additionally, all except logger can be configured with environment variables (see above, but the pattern is like: "SNOWFLAKE_HTTP_RETRIES". Configuration options can only be set on initiialization through `new` or `from_env`.
|
83
109
|
|
84
|
-
`logger` - takes any ruby logger (by default it's a std lib Logger.new(STDOUT), set at DEBUG level. Not available as an ENV variable config option
|
85
|
-
`log_level` - takes a log level, type is dependent on logger, for the default ruby Logger, use a level like `Logger::WARN`. Not available as an ENV variable config option.
|
86
|
-
`jwt_token_ttl` - The time to live set on JWT token in seconds, defaults to 3540 (59 minutes, the longest Snowflake supports is 60)
|
87
|
-
`connection_timeout` - The amount of time in seconds that the client's connection pool will wait before erroring in handing out a valid connection, defaults to 60 seconds
|
88
|
-
`max_connections` - The maximum number of http connections to hold open in the connection pool. If you use the client in a threaded context, you may need to increase this to be threads * client.max_threads_per_query, defaults to 16.
|
89
|
-
`max_threads_per_query` - The maximum number of threads the client should use to retreive data, per query, defaults to 8. If you want the client to act in a single threaded way, set this to 1
|
90
|
-
`thread_scale_factor` - When downloading a result set into memory, thread count is calculated by dividing a query's partition count by this number. For details on implementation see the code in `client.rb`.
|
91
|
-
`http_retries` - By default the client will retry common typically transient errors (http responses) twice, you can change the number of retries with this.
|
110
|
+
- `logger` - takes any ruby logger (by default it's a std lib Logger.new(STDOUT), set at DEBUG level. Not available as an ENV variable config option
|
111
|
+
- `log_level` - takes a log level, type is dependent on logger, for the default ruby Logger, use a level like `Logger::WARN`. Not available as an ENV variable config option.
|
112
|
+
- `jwt_token_ttl` - The time to live set on JWT token in seconds, defaults to 3540 (59 minutes, the longest Snowflake supports is 60).
|
113
|
+
- `connection_timeout` - The amount of time in seconds that the client's connection pool will wait before erroring in handing out a valid connection, defaults to 60 seconds
|
114
|
+
- `max_connections` - The maximum number of http connections to hold open in the connection pool. If you use the client in a threaded context, you may need to increase this to be threads * client.max_threads_per_query, defaults to 16.
|
115
|
+
- `max_threads_per_query` - The maximum number of threads the client should use to retreive data, per query, defaults to 8. If you want the client to act in a single threaded way, set this to 1
|
116
|
+
- `thread_scale_factor` - When downloading a result set into memory, thread count is calculated by dividing a query's partition count by this number. For details on implementation see the code in `client.rb`.
|
117
|
+
- `http_retries` - By default the client will retry common typically transient errors (http responses) twice, you can change the number of retries with this.
|
92
118
|
|
93
119
|
Example configuration:
|
94
120
|
```ruby
|
95
|
-
client = RubySnowflake::Client.from_env
|
96
|
-
|
97
|
-
|
98
|
-
|
121
|
+
client = RubySnowflake::Client.from_env(
|
122
|
+
logger: Rails.logger
|
123
|
+
max_connections: 24
|
124
|
+
http_retries 1
|
125
|
+
)
|
99
126
|
end
|
127
|
+
```
|
100
128
|
|
101
129
|
# Gotchas
|
102
130
|
|
@@ -153,6 +181,7 @@ client = RubySnowflake::Client.new(
|
|
153
181
|
"snowflake-account", # typically your subdomain
|
154
182
|
"snowflake-user", # Your snowflake user
|
155
183
|
"some_warehouse", # The name of your warehouse to use by default
|
184
|
+
"some_database", # The name of the database in the context of which the queries will run
|
156
185
|
)
|
157
186
|
```
|
158
187
|
|
@@ -0,0 +1,59 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "jwt"
|
4
|
+
require "openssl"
|
5
|
+
require "concurrent"
|
6
|
+
|
7
|
+
module RubySnowflake
|
8
|
+
class Client
|
9
|
+
class KeyPairJwtAuthManager
|
10
|
+
# requires text of a PEM formatted RSA private key
|
11
|
+
def initialize(organization, account, user, private_key, jwt_token_ttl)
|
12
|
+
@organization = organization
|
13
|
+
@account = account
|
14
|
+
@user = user
|
15
|
+
@private_key_pem = private_key
|
16
|
+
@jwt_token_ttl = jwt_token_ttl
|
17
|
+
|
18
|
+
# start with an expired value to force creation
|
19
|
+
@token_expires_at = Time.now.to_i - 1
|
20
|
+
@token_semaphore = Concurrent::Semaphore.new(1)
|
21
|
+
end
|
22
|
+
|
23
|
+
def jwt_token
|
24
|
+
return @token unless jwt_token_expired?
|
25
|
+
|
26
|
+
@token_semaphore.acquire do
|
27
|
+
now = Time.now.to_i
|
28
|
+
@token_expires_at = now + @jwt_token_ttl
|
29
|
+
|
30
|
+
private_key = OpenSSL::PKey.read(@private_key_pem)
|
31
|
+
|
32
|
+
payload = {
|
33
|
+
:iss => "#{@organization.upcase}-#{@account.upcase}.#{@user}.#{public_key_fingerprint}",
|
34
|
+
:sub => "#{@organization.upcase}-#{@account.upcase}.#{@user}",
|
35
|
+
:iat => now,
|
36
|
+
:exp => @token_expires_at
|
37
|
+
}
|
38
|
+
|
39
|
+
@token = JWT.encode payload, private_key, "RS256"
|
40
|
+
end
|
41
|
+
end
|
42
|
+
|
43
|
+
private
|
44
|
+
def jwt_token_expired?
|
45
|
+
Time.now.to_i > @token_expires_at
|
46
|
+
end
|
47
|
+
|
48
|
+
def public_key_fingerprint
|
49
|
+
reutrn @public_key_fingerprint unless @public_key_fingerprint.nil?
|
50
|
+
|
51
|
+
public_key_der = OpenSSL::PKey::RSA.new(@private_key_pem).public_key.to_der
|
52
|
+
digest = OpenSSL::Digest::SHA256.new.digest(public_key_der)
|
53
|
+
fingerprint = Base64.strict_encode64(digest)
|
54
|
+
|
55
|
+
@public_key_fingerprint = "SHA256:#{fingerprint}"
|
56
|
+
end
|
57
|
+
end
|
58
|
+
end
|
59
|
+
end
|
@@ -5,22 +5,21 @@ require "benchmark"
|
|
5
5
|
require "concurrent"
|
6
6
|
require "connection_pool"
|
7
7
|
require "json"
|
8
|
-
require "jwt"
|
9
8
|
require "logger"
|
10
9
|
require "net/http"
|
11
10
|
require "oj"
|
12
|
-
require "openssl"
|
13
11
|
require "retryable"
|
14
12
|
require "securerandom"
|
15
13
|
require "uri"
|
16
14
|
|
17
15
|
|
18
|
-
require_relative "result"
|
19
|
-
require_relative "streaming_result"
|
20
16
|
require_relative "client/http_connection_wrapper"
|
17
|
+
require_relative "client/key_pair_jwt_auth_manager"
|
21
18
|
require_relative "client/single_thread_in_memory_strategy"
|
22
19
|
require_relative "client/streaming_result_strategy"
|
23
20
|
require_relative "client/threaded_in_memory_strategy"
|
21
|
+
require_relative "result"
|
22
|
+
require_relative "streaming_result"
|
24
23
|
|
25
24
|
module RubySnowflake
|
26
25
|
class Error < StandardError
|
@@ -55,11 +54,19 @@ module RubySnowflake
|
|
55
54
|
# how many times to retry common retryable HTTP responses (i.e. 429, 504)
|
56
55
|
DEFAULT_HTTP_RETRIES = 2
|
57
56
|
|
58
|
-
|
59
|
-
attr_reader :connection_timeout, :max_connections
|
60
|
-
attr_accessor :logger, :jwt_token_ttl, :max_threads_per_query, :thread_scale_factor, :http_retries
|
57
|
+
OJ_OPTIONS = { :bigdecimal_load => :bigdecimal }.freeze
|
61
58
|
|
62
|
-
|
59
|
+
# can't be set after initialization
|
60
|
+
attr_reader :connection_timeout, :max_connections, :logger, :max_threads_per_query, :thread_scale_factor, :http_retries
|
61
|
+
|
62
|
+
def self.from_env(logger: DEFAULT_LOGGER,
|
63
|
+
log_level: DEFAULT_LOG_LEVEL,
|
64
|
+
jwt_token_ttl: env_option("SNOWFLAKE_JWT_TOKEN_TTL", DEFAULT_JWT_TOKEN_TTL),
|
65
|
+
connection_timeout: env_option("SNOWFLAKE_CONNECTION_TIMEOUT", DEFAULT_CONNECTION_TIMEOUT ),
|
66
|
+
max_connections: env_option("SNOWFLAKE_MAX_CONNECTIONS", DEFAULT_MAX_CONNECTIONS ),
|
67
|
+
max_threads_per_query: env_option("SNOWFLAKE_MAX_THREADS_PER_QUERY", DEFAULT_MAX_THREADS_PER_QUERY),
|
68
|
+
thread_scale_factor: env_option("SNOWFLAKE_THREAD_SCALE_FACTOR", DEFAULT_THREAD_SCALE_FACTOR),
|
69
|
+
http_retries: env_option("SNOWFLAKE_HTTP_RETRIES", DEFAULT_HTTP_RETRIES))
|
63
70
|
private_key = ENV["SNOWFLAKE_PRIVATE_KEY"] || File.read(ENV["SNOWFLAKE_PRIVATE_KEY_PATH"])
|
64
71
|
|
65
72
|
new(
|
@@ -69,53 +76,54 @@ module RubySnowflake
|
|
69
76
|
ENV["SNOWFLAKE_ACCOUNT"],
|
70
77
|
ENV["SNOWFLAKE_USER"],
|
71
78
|
ENV["SNOWFLAKE_DEFAULT_WAREHOUSE"],
|
72
|
-
|
73
|
-
|
74
|
-
|
75
|
-
|
76
|
-
|
77
|
-
|
79
|
+
ENV["SNOWFLAKE_DEFAULT_DATABASE"],
|
80
|
+
logger: logger,
|
81
|
+
log_level: log_level,
|
82
|
+
jwt_token_ttl: jwt_token_ttl,
|
83
|
+
connection_timeout: connection_timeout,
|
84
|
+
max_connections: max_connections,
|
85
|
+
max_threads_per_query: max_threads_per_query,
|
86
|
+
thread_scale_factor: thread_scale_factor,
|
87
|
+
http_retries: http_retries,
|
78
88
|
)
|
79
89
|
end
|
80
90
|
|
81
|
-
def initialize(
|
82
|
-
|
83
|
-
|
84
|
-
|
85
|
-
|
86
|
-
|
87
|
-
|
88
|
-
|
89
|
-
|
91
|
+
def initialize(
|
92
|
+
uri, private_key, organization, account, user, default_warehouse, default_database,
|
93
|
+
logger: DEFAULT_LOGGER,
|
94
|
+
log_level: DEFAULT_LOG_LEVEL,
|
95
|
+
jwt_token_ttl: DEFAULT_JWT_TOKEN_TTL,
|
96
|
+
connection_timeout: DEFAULT_CONNECTION_TIMEOUT,
|
97
|
+
max_connections: DEFAULT_MAX_CONNECTIONS,
|
98
|
+
max_threads_per_query: DEFAULT_MAX_THREADS_PER_QUERY,
|
99
|
+
thread_scale_factor: DEFAULT_THREAD_SCALE_FACTOR,
|
100
|
+
http_retries: DEFAULT_HTTP_RETRIES
|
101
|
+
)
|
90
102
|
@base_uri = uri
|
91
|
-
@
|
92
|
-
|
93
|
-
@account = account
|
94
|
-
@user = user
|
103
|
+
@key_pair_jwt_auth_manager =
|
104
|
+
KeyPairJwtAuthManager.new(organization, account, user, private_key, jwt_token_ttl)
|
95
105
|
@default_warehouse = default_warehouse
|
96
|
-
@
|
106
|
+
@default_database = default_database
|
97
107
|
|
98
108
|
# set defaults for config settings
|
99
109
|
@logger = logger
|
100
110
|
@logger.level = log_level
|
101
|
-
@jwt_token_ttl = jwt_token_ttl
|
102
111
|
@connection_timeout = connection_timeout
|
103
112
|
@max_connections = max_connections
|
104
113
|
@max_threads_per_query = max_threads_per_query
|
105
114
|
@thread_scale_factor = thread_scale_factor
|
106
115
|
@http_retries = http_retries
|
107
|
-
|
108
|
-
# start with an expired value to force creation
|
109
|
-
@token_expires_at = Time.now.to_i - 1
|
110
|
-
@token_semaphore = Concurrent::Semaphore.new(1)
|
111
116
|
end
|
112
117
|
|
113
|
-
def query(query, warehouse: nil, streaming: false)
|
118
|
+
def query(query, warehouse: nil, streaming: false, database: nil)
|
114
119
|
warehouse ||= @default_warehouse
|
120
|
+
database ||= @default_database
|
115
121
|
|
116
122
|
response = nil
|
117
123
|
connection_pool.with do |connection|
|
118
|
-
request_body = {
|
124
|
+
request_body = {
|
125
|
+
"statement" => query, "warehouse" => warehouse, "database" => database
|
126
|
+
}
|
119
127
|
|
120
128
|
response = request_with_auth_and_headers(
|
121
129
|
connection,
|
@@ -150,36 +158,19 @@ module RubySnowflake
|
|
150
158
|
@port ||= URI.parse(@base_uri).port
|
151
159
|
end
|
152
160
|
|
153
|
-
def
|
154
|
-
|
155
|
-
|
156
|
-
|
157
|
-
now = Time.now.to_i
|
158
|
-
@token_expires_at = now + @jwt_token_ttl
|
159
|
-
|
160
|
-
private_key = OpenSSL::PKey.read(@private_key_pem)
|
161
|
-
|
162
|
-
payload = {
|
163
|
-
:iss => "#{@organization.upcase}-#{@account.upcase}.#{@user}.#{@public_key_fingerprint}",
|
164
|
-
:sub => "#{@organization.upcase}-#{@account.upcase}.#{@user}",
|
165
|
-
:iat => now,
|
166
|
-
:exp => @token_expires_at
|
167
|
-
}
|
168
|
-
|
169
|
-
@token = JWT.encode payload, private_key, "RS256"
|
161
|
+
def handle_errors(response)
|
162
|
+
if response.code != "200"
|
163
|
+
raise BadResponseError.new({}),
|
164
|
+
"Bad response! Got code: #{response.code}, w/ message #{response.body}"
|
170
165
|
end
|
171
166
|
end
|
172
167
|
|
173
|
-
def jwt_token_expired?
|
174
|
-
Time.now.to_i > @token_expires_at
|
175
|
-
end
|
176
|
-
|
177
168
|
def request_with_auth_and_headers(connection, request_class, path, body=nil)
|
178
169
|
uri = URI.parse("#{@base_uri}#{path}")
|
179
170
|
request = request_class.new(uri)
|
180
171
|
request["Content-Type"] = "application/json"
|
181
172
|
request["Accept"] = "application/json"
|
182
|
-
request["Authorization"] = "Bearer #{jwt_token}"
|
173
|
+
request["Authorization"] = "Bearer #{@key_pair_jwt_auth_manager.jwt_token}"
|
183
174
|
request["X-Snowflake-Authorization-Token-Type"] = "KEYPAIR_JWT"
|
184
175
|
request.body = body unless body.nil?
|
185
176
|
|
@@ -223,7 +214,7 @@ module RubySnowflake
|
|
223
214
|
end
|
224
215
|
|
225
216
|
def retreive_result_set(response, streaming)
|
226
|
-
json_body = Oj.load(response.body,
|
217
|
+
json_body = Oj.load(response.body, OJ_OPTIONS)
|
227
218
|
statement_handle = json_body["statementHandle"]
|
228
219
|
num_threads = number_of_threads_to_use(json_body["resultSetMetaData"]["partitionInfo"].size)
|
229
220
|
retreive_proc = ->(index) { retreive_partition_data(statement_handle, index) }
|
@@ -247,8 +238,8 @@ module RubySnowflake
|
|
247
238
|
)
|
248
239
|
end
|
249
240
|
|
250
|
-
partition_json =
|
251
|
-
bm = Benchmark.measure { partition_json = Oj.load(partition_response.body,
|
241
|
+
partition_json = {}
|
242
|
+
bm = Benchmark.measure { partition_json = Oj.load(partition_response.body, OJ_OPTIONS) }
|
252
243
|
logger.debug { "JSON parsing took: #{bm.real}" }
|
253
244
|
partition_data = partition_json["data"]
|
254
245
|
|
@@ -258,17 +249,5 @@ module RubySnowflake
|
|
258
249
|
def number_of_threads_to_use(partition_count)
|
259
250
|
[[1, (partition_count / @thread_scale_factor.to_f).ceil].max, @max_threads_per_query].min
|
260
251
|
end
|
261
|
-
|
262
|
-
def oj_options
|
263
|
-
{ :bigdecimal_load => :bigdecimal }
|
264
|
-
end
|
265
|
-
|
266
|
-
def public_key_fingerprint(private_key_pem_string)
|
267
|
-
public_key_der = OpenSSL::PKey::RSA.new(private_key_pem_string).public_key.to_der
|
268
|
-
digest = OpenSSL::Digest::SHA256.new.digest(public_key_der)
|
269
|
-
fingerprint = Base64.strict_encode64(digest)
|
270
|
-
|
271
|
-
"SHA256:#{fingerprint}"
|
272
|
-
end
|
273
252
|
end
|
274
253
|
end
|
data/lib/ruby_snowflake/row.rb
CHANGED
@@ -31,14 +31,19 @@ module RubySnowflake
|
|
31
31
|
else
|
32
32
|
BigDecimal(@data[index]).round(@row_types[index][:scale])
|
33
33
|
end
|
34
|
+
|
35
|
+
# snowflake treats these all as 64 bit IEEE 754 floating point numbers, and will we too
|
34
36
|
when :float, :double, :"double precision", :real
|
35
|
-
# snowflake treats these all as 64 bit IEEE 754 floating point numbers, and will we too
|
36
37
|
Float(@data[index])
|
37
|
-
|
38
|
+
|
39
|
+
# Despite snowflake indicating that it sends the offset in minutes, the actual time in UTC
|
40
|
+
# is always sent in the first half of the data. If an offset is sent it looks like:
|
41
|
+
# "1641008096.123000000 1980"
|
42
|
+
# If there isn't one, it's just like this:
|
43
|
+
# "1641065696.123000000"
|
44
|
+
# in all cases, the actual time, in UTC is the float value, and the offset is ignorable
|
45
|
+
when :time, :datetime, :timestamp, :timestamp_ntz, :timestamp_ltz, :timestamp_tz
|
38
46
|
Time.strptime(@data[index], TIME_FORMAT).utc
|
39
|
-
when :timestamp_tz
|
40
|
-
timestamp, offset_minutes = @data[index].split(" ")
|
41
|
-
Time.strptime(@data[index], TIME_FORMAT).utc - (Integer(offset_minutes) * 60)
|
42
47
|
else
|
43
48
|
@data[index]
|
44
49
|
end
|
data/rb_snowflake_client.gemspec
CHANGED
@@ -18,4 +18,10 @@ Gem::Specification.new do |s|
|
|
18
18
|
end
|
19
19
|
|
20
20
|
s.require_paths = ["lib"]
|
21
|
+
s.add_dependency "concurrent-ruby", ">= 1.2"
|
22
|
+
s.add_dependency "connection_pool", ">= 2.4"
|
23
|
+
s.add_dependency "dotenv", ">= 2.8"
|
24
|
+
s.add_dependency "jwt", ">= 2.7"
|
25
|
+
s.add_dependency "oj", ">= 3.16"
|
26
|
+
s.add_dependency "retryable", ">= 3.0"
|
21
27
|
end
|
metadata
CHANGED
@@ -1,15 +1,99 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: rb_snowflake_client
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.2.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Rinsed
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2023-12-
|
12
|
-
dependencies:
|
11
|
+
date: 2023-12-07 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: concurrent-ruby
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - ">="
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: '1.2'
|
20
|
+
type: :runtime
|
21
|
+
prerelease: false
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - ">="
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: '1.2'
|
27
|
+
- !ruby/object:Gem::Dependency
|
28
|
+
name: connection_pool
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - ">="
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: '2.4'
|
34
|
+
type: :runtime
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - ">="
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: '2.4'
|
41
|
+
- !ruby/object:Gem::Dependency
|
42
|
+
name: dotenv
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - ">="
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '2.8'
|
48
|
+
type: :runtime
|
49
|
+
prerelease: false
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - ">="
|
53
|
+
- !ruby/object:Gem::Version
|
54
|
+
version: '2.8'
|
55
|
+
- !ruby/object:Gem::Dependency
|
56
|
+
name: jwt
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - ">="
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '2.7'
|
62
|
+
type: :runtime
|
63
|
+
prerelease: false
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - ">="
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: '2.7'
|
69
|
+
- !ruby/object:Gem::Dependency
|
70
|
+
name: oj
|
71
|
+
requirement: !ruby/object:Gem::Requirement
|
72
|
+
requirements:
|
73
|
+
- - ">="
|
74
|
+
- !ruby/object:Gem::Version
|
75
|
+
version: '3.16'
|
76
|
+
type: :runtime
|
77
|
+
prerelease: false
|
78
|
+
version_requirements: !ruby/object:Gem::Requirement
|
79
|
+
requirements:
|
80
|
+
- - ">="
|
81
|
+
- !ruby/object:Gem::Version
|
82
|
+
version: '3.16'
|
83
|
+
- !ruby/object:Gem::Dependency
|
84
|
+
name: retryable
|
85
|
+
requirement: !ruby/object:Gem::Requirement
|
86
|
+
requirements:
|
87
|
+
- - ">="
|
88
|
+
- !ruby/object:Gem::Version
|
89
|
+
version: '3.0'
|
90
|
+
type: :runtime
|
91
|
+
prerelease: false
|
92
|
+
version_requirements: !ruby/object:Gem::Requirement
|
93
|
+
requirements:
|
94
|
+
- - ">="
|
95
|
+
- !ruby/object:Gem::Version
|
96
|
+
version: '3.0'
|
13
97
|
description: 'Using the HTTP V2 Api for Snowflake runs queries & creates native Ruby
|
14
98
|
objects.
|
15
99
|
|
@@ -33,6 +117,7 @@ files:
|
|
33
117
|
- lib/rb_snowflake_client.rb
|
34
118
|
- lib/ruby_snowflake/client.rb
|
35
119
|
- lib/ruby_snowflake/client/http_connection_wrapper.rb
|
120
|
+
- lib/ruby_snowflake/client/key_pair_jwt_auth_manager.rb
|
36
121
|
- lib/ruby_snowflake/client/single_thread_in_memory_strategy.rb
|
37
122
|
- lib/ruby_snowflake/client/streaming_result_strategy.rb
|
38
123
|
- lib/ruby_snowflake/client/threaded_in_memory_strategy.rb
|