olaf 0.1.2 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +91 -10
- data/lib/olaf/drivers/big_query.rb +104 -0
- data/lib/olaf/drivers/snowflake.rb +12 -1
- data/lib/olaf/errors.rb +12 -0
- data/lib/olaf/query_definition/class_methods.rb +29 -0
- data/lib/olaf/query_definition.rb +14 -7
- data/lib/olaf.rb +56 -19
- data/olaf.gemspec +8 -4
- data/test/olaf_configure_test.rb +66 -0
- data/test/olaf_execute_test.rb +66 -5
- data/test/olaf_lazy_loading_test.rb +40 -0
- metadata +16 -17
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 4d1640a5221b0ed5a71b4fbb2ca11d113d69ef7436e7b7c2628497c4767a4245
|
|
4
|
+
data.tar.gz: fd816660fe86290c5f5d8148a11823b2c2580e66096817d343b958fcbb1fc30e
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: b1f71e253ff7cb95a7eef3d97324f6615d43c57aaa39e9128fcdce5f8ef683ed2a742d852aff034adf3cbd8fd04ddfa2d58a1c7372edea79843f0ab12581735a
|
|
7
|
+
data.tar.gz: db997081fd2f622d354cd7bcb65a73c45ef55eba1bb71d8788b08590cb02e3500cdc428da78697ff838666daac3996d6f8a83cdfd14bbd3750655e308967507c
|
data/README.md
CHANGED
|
@@ -1,21 +1,21 @@
|
|
|
1
1
|
# Olaf
|
|
2
|
-
[](https://badge.fury.io/rb/olaf)
|
|
3
|
-
[](https://circleci.com/gh/carwow/olaf)
|
|
4
2
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
Olaf is a small Ruby wrapper for Snowflake queries.
|
|
3
|
+
Olaf is a small Ruby wrapper for warehouse queries, running them in Snowflake or
|
|
4
|
+
in BigQuery.
|
|
8
5
|
|
|
9
6
|

|
|
10
7
|
|
|
11
8
|
|
|
12
9
|
## Dependencies
|
|
13
10
|
|
|
14
|
-
`olaf`
|
|
15
|
-
|
|
16
|
-
Install dependencies using `bundler` is easy as run:
|
|
11
|
+
`olaf` depends on nothing by itself. Each driver loads its own client library,
|
|
12
|
+
and only when it is used, so add to your `Gemfile` the ones you configure:
|
|
17
13
|
|
|
18
|
-
|
|
14
|
+
| Driver | Gems | Also needs |
|
|
15
|
+
| ----------------- | ----------------------- | ----------------------- |
|
|
16
|
+
| `Olaf::Snowflake` | `sequel`, `ruby-odbc` | the ODBC system library |
|
|
17
|
+
| `Olaf::BigQuery` | `google-cloud-bigquery` | |
|
|
18
|
+
| `Olaf::Fake` | — | |
|
|
19
19
|
|
|
20
20
|
## Installation
|
|
21
21
|
|
|
@@ -25,7 +25,7 @@ If you don't have Olaf, try this:
|
|
|
25
25
|
|
|
26
26
|
## Getting started
|
|
27
27
|
|
|
28
|
-
Olaf helps developers to represent
|
|
28
|
+
Olaf helps developers to represent warehouse queries as objects, to have more
|
|
29
29
|
control in the code and in tests.
|
|
30
30
|
|
|
31
31
|
### Example
|
|
@@ -46,3 +46,84 @@ query = FetchUsers.prepare(department_id: 1337)
|
|
|
46
46
|
Olaf.execute(query)
|
|
47
47
|
=> [#<User id: 41, department_id: 1337, name: 'Ian'>]
|
|
48
48
|
```
|
|
49
|
+
|
|
50
|
+
## Configuration
|
|
51
|
+
|
|
52
|
+
One driver, which every query runs on:
|
|
53
|
+
|
|
54
|
+
```ruby
|
|
55
|
+
Olaf.configure(user: 'olaf') # Olaf::Snowflake, the default
|
|
56
|
+
Olaf.configure(olaf_driver: Olaf::Fake) # ideal for testing
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
Or several, which queries pick by name. The ones that don't declare a `driver`
|
|
60
|
+
run on the `default:`, the first one given when it is not specified:
|
|
61
|
+
|
|
62
|
+
```ruby
|
|
63
|
+
Olaf.configure(
|
|
64
|
+
drivers: {
|
|
65
|
+
snowflake: Olaf::Snowflake.new(user: 'olaf'),
|
|
66
|
+
big_query: Olaf::BigQuery.new(project: 'carwow', labels: { service: 'flatmin' })
|
|
67
|
+
},
|
|
68
|
+
default: :snowflake
|
|
69
|
+
)
|
|
70
|
+
|
|
71
|
+
class FetchUsers
|
|
72
|
+
include Olaf::QueryDefinition
|
|
73
|
+
|
|
74
|
+
driver :big_query
|
|
75
|
+
|
|
76
|
+
template './big_query/users_in_department.sql'
|
|
77
|
+
|
|
78
|
+
argument :department_id
|
|
79
|
+
end
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
`Olaf.instance(:big_query)` returns that driver. A single configured driver
|
|
83
|
+
serves every query, whatever they declare, which is what keeps `Olaf::Fake`
|
|
84
|
+
covering all of them in tests:
|
|
85
|
+
|
|
86
|
+
```ruby
|
|
87
|
+
Olaf.configure(olaf_driver: Olaf::Fake)
|
|
88
|
+
|
|
89
|
+
Olaf.instance.register_result(FetchUsers, [{ id: 41 }])
|
|
90
|
+
Olaf.instance.register_result(FetchUsers, [{ id: 42 }], with: { department_id: 1337 })
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
Whichever driver runs the query, a failure it reports is wrapped into
|
|
94
|
+
`Olaf::QueryExecutionError`, carrying the query metadata.
|
|
95
|
+
|
|
96
|
+
## BigQuery
|
|
97
|
+
|
|
98
|
+
```ruby
|
|
99
|
+
Olaf::BigQuery.new(
|
|
100
|
+
project: 'carwow',
|
|
101
|
+
credentials: JSON.parse(ENV['BIGQUERY_CREDENTIALS']), # ambient Google credentials when omitted
|
|
102
|
+
maximum_bytes_billed: 5 * Olaf::BigQuery::GIGABYTE, # default: 1 GiB
|
|
103
|
+
labels: { service: 'flatmin', country: 'uk' }
|
|
104
|
+
)
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
The query runs as a job, so that it can be capped and labelled:
|
|
108
|
+
|
|
109
|
+
* Olaf's `:placeholders` are rewritten to BigQuery's `@named` parameters, for
|
|
110
|
+
declared arguments only, and bound as query parameters. Arguments declared
|
|
111
|
+
`as: :literal` are substituted into the SQL before that, as usual.
|
|
112
|
+
* Every job is capped with `maximum_bytes_billed`, and labelled with the driver
|
|
113
|
+
labels plus the name of the template. A query that scans more than the cap
|
|
114
|
+
raises its own, which keeps a regression failing instead of quietly costing
|
|
115
|
+
money:
|
|
116
|
+
|
|
117
|
+
```ruby
|
|
118
|
+
driver :big_query, maximum_bytes_billed: 10 * Olaf::BigQuery::GIGABYTE
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
`Olaf.execute` returns the rows. `Olaf.instance(:big_query).run(query)` returns
|
|
122
|
+
the finished job instead, for its statistics:
|
|
123
|
+
|
|
124
|
+
```ruby
|
|
125
|
+
job = Olaf.instance(:big_query).run(query)
|
|
126
|
+
job.bytes_processed
|
|
127
|
+
job.cache_hit?
|
|
128
|
+
job.data.all.to_a
|
|
129
|
+
```
|
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
# Not a dependency of the gem: consumers of this driver add it themselves.
|
|
2
|
+
require 'google/cloud/bigquery'
|
|
3
|
+
|
|
4
|
+
# The API client debug-logs whole response bodies, which for a warehouse query
|
|
5
|
+
# is the entire result set.
|
|
6
|
+
Google::Apis.logger.level = [Google::Apis.logger.level, Logger::INFO].max
|
|
7
|
+
|
|
8
|
+
module Olaf
|
|
9
|
+
class BigQuery
|
|
10
|
+
GIGABYTE = 1024**3
|
|
11
|
+
|
|
12
|
+
# Restrictive on purpose: a query that scans more says so with
|
|
13
|
+
# `driver :big_query, maximum_bytes_billed: ...`.
|
|
14
|
+
DEFAULT_MAXIMUM_BYTES_BILLED = GIGABYTE
|
|
15
|
+
|
|
16
|
+
# Labels only accept lowercase letters, numbers, dashes and underscores.
|
|
17
|
+
LABEL_FORMAT = /[^a-z0-9_-]/.freeze
|
|
18
|
+
LABEL_MAX_LENGTH = 63
|
|
19
|
+
|
|
20
|
+
# Credentials default to the ambient Google ones, and the client is built
|
|
21
|
+
# from the arguments unless one is given.
|
|
22
|
+
def initialize(project: nil, credentials: nil, maximum_bytes_billed: DEFAULT_MAXIMUM_BYTES_BILLED,
|
|
23
|
+
labels: {}, client: nil)
|
|
24
|
+
@project = project
|
|
25
|
+
@credentials = credentials
|
|
26
|
+
@maximum_bytes_billed = maximum_bytes_billed
|
|
27
|
+
@labels = labels
|
|
28
|
+
@client = client
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
def fetch(olaf_query)
|
|
32
|
+
run(olaf_query).data.all.to_a
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
# Returns the finished job, so callers can read its stats (bytes billed,
|
|
36
|
+
# cache hit) on top of its rows.
|
|
37
|
+
#
|
|
38
|
+
# @raises Olaf::QueryExecutionError
|
|
39
|
+
def run(olaf_query)
|
|
40
|
+
job = client.query_job(
|
|
41
|
+
sql(olaf_query),
|
|
42
|
+
params: params(olaf_query),
|
|
43
|
+
maximum_bytes_billed: maximum_bytes_billed(olaf_query),
|
|
44
|
+
labels: labels(olaf_query)
|
|
45
|
+
)
|
|
46
|
+
job.wait_until_done!
|
|
47
|
+
|
|
48
|
+
raise QueryExecutionError.new(job.error.to_h['message'].to_s, olaf_query) if job.failed?
|
|
49
|
+
|
|
50
|
+
job
|
|
51
|
+
rescue *wrapped_errors => error
|
|
52
|
+
raise QueryExecutionError.new(error.message, olaf_query)
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
private
|
|
56
|
+
|
|
57
|
+
def client
|
|
58
|
+
@client ||= Google::Cloud::Bigquery.new(project: @project, credentials: @credentials)
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
# Google::Cloud::Error covers everything the API reports, Google::Auth::Error
|
|
62
|
+
# credentials missing or expired before a call is made. The second one only
|
|
63
|
+
# exists in googleauth 1.14 and later.
|
|
64
|
+
def wrapped_errors
|
|
65
|
+
@wrapped_errors ||= [
|
|
66
|
+
Google::Cloud::Error,
|
|
67
|
+
(Google::Auth::Error if defined?(Google::Auth::Error))
|
|
68
|
+
].compact
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
# olaf validates `:foo` placeholders, BigQuery binds `@foo`. Only declared
|
|
72
|
+
# argument names are rewritten, so casts and time literals are left alone.
|
|
73
|
+
def sql(olaf_query)
|
|
74
|
+
params(olaf_query).keys.reduce(olaf_query.sql_template) do |sql, name|
|
|
75
|
+
sql.gsub(/(?<!:):#{name}\b/, "@#{name}")
|
|
76
|
+
end
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
# Literal arguments were substituted into the template by `prepare`.
|
|
80
|
+
def params(olaf_query)
|
|
81
|
+
literal_arguments = literal_arguments(olaf_query)
|
|
82
|
+
|
|
83
|
+
olaf_query.variables.reject { |name, _value| literal_arguments.include?(name) }
|
|
84
|
+
end
|
|
85
|
+
|
|
86
|
+
def literal_arguments(olaf_query)
|
|
87
|
+
olaf_query.class.arguments.select { |_name, options| options[:literal] }.keys
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
def maximum_bytes_billed(olaf_query)
|
|
91
|
+
olaf_query.class.driver_options.fetch(:maximum_bytes_billed, @maximum_bytes_billed)
|
|
92
|
+
end
|
|
93
|
+
|
|
94
|
+
def labels(olaf_query)
|
|
95
|
+
@labels
|
|
96
|
+
.merge(query: File.basename(olaf_query.class.template.to_s, '.sql'))
|
|
97
|
+
.transform_values { |value| label_value(value) }
|
|
98
|
+
end
|
|
99
|
+
|
|
100
|
+
def label_value(value)
|
|
101
|
+
value.to_s.downcase.gsub(LABEL_FORMAT, '_')[0, LABEL_MAX_LENGTH]
|
|
102
|
+
end
|
|
103
|
+
end
|
|
104
|
+
end
|
|
@@ -1,17 +1,28 @@
|
|
|
1
|
+
require 'sequel'
|
|
2
|
+
|
|
1
3
|
module Olaf
|
|
2
4
|
class Snowflake
|
|
5
|
+
WRAPPED_ERRORS = [Sequel::DatabaseError].freeze
|
|
6
|
+
|
|
3
7
|
def initialize(**config)
|
|
4
8
|
@config = config
|
|
5
9
|
end
|
|
6
10
|
|
|
7
11
|
def fetch(olaf_query)
|
|
8
12
|
conn.fetch(olaf_query.sql_template, **olaf_query.variables).all
|
|
13
|
+
rescue *WRAPPED_ERRORS => error
|
|
14
|
+
raise QueryExecutionError.new(error.message, olaf_query)
|
|
9
15
|
end
|
|
10
16
|
|
|
11
17
|
private
|
|
12
18
|
|
|
13
19
|
def conn
|
|
14
|
-
|
|
20
|
+
# Only opening a connection needs the ODBC system library.
|
|
21
|
+
@conn ||= begin
|
|
22
|
+
require 'odbc_utf8'
|
|
23
|
+
|
|
24
|
+
Sequel.odbc('snowflake', **@config)
|
|
25
|
+
end
|
|
15
26
|
end
|
|
16
27
|
end
|
|
17
28
|
end
|
data/lib/olaf/errors.rb
CHANGED
|
@@ -23,6 +23,18 @@ module Olaf
|
|
|
23
23
|
end
|
|
24
24
|
end
|
|
25
25
|
|
|
26
|
+
class UnknownArgumentsError < StandardError
|
|
27
|
+
def initialize(olaf_query)
|
|
28
|
+
super("Unknown arguments: #{olaf_query.unknown_arguments}")
|
|
29
|
+
end
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
class UnknownDriverError < StandardError
|
|
33
|
+
def initialize(name, configured_drivers)
|
|
34
|
+
super("Unknown driver #{name.inspect}, configured: #{configured_drivers.inspect}")
|
|
35
|
+
end
|
|
36
|
+
end
|
|
37
|
+
|
|
26
38
|
class QueryExecutionError < StandardError
|
|
27
39
|
attr_reader :metadata
|
|
28
40
|
|
|
@@ -58,6 +58,35 @@ module Olaf
|
|
|
58
58
|
@template ||= file_name
|
|
59
59
|
end
|
|
60
60
|
|
|
61
|
+
# Define the driver this query runs on, one of the names given to
|
|
62
|
+
# `Olaf.configure`. Defaults to the default driver.
|
|
63
|
+
#
|
|
64
|
+
# Options are driver specific: the BigQuery driver reads
|
|
65
|
+
# :maximum_bytes_billed.
|
|
66
|
+
#
|
|
67
|
+
# Example:
|
|
68
|
+
#
|
|
69
|
+
# class OneQuery
|
|
70
|
+
# include Olaf::QueryDefinition
|
|
71
|
+
#
|
|
72
|
+
# driver :big_query, maximum_bytes_billed: 5 * Olaf::BigQuery::GIGABYTE
|
|
73
|
+
# end
|
|
74
|
+
#
|
|
75
|
+
#
|
|
76
|
+
def driver(name = nil, **options)
|
|
77
|
+
return @driver if name.nil?
|
|
78
|
+
|
|
79
|
+
@driver ||= name
|
|
80
|
+
@driver_options ||= options
|
|
81
|
+
|
|
82
|
+
@driver
|
|
83
|
+
end
|
|
84
|
+
|
|
85
|
+
# @return Hash of the options given to `driver`
|
|
86
|
+
def driver_options
|
|
87
|
+
@driver_options ||= {}
|
|
88
|
+
end
|
|
89
|
+
|
|
61
90
|
# Define the object representing each row of the result.
|
|
62
91
|
# When not specified, each row will be a hash by default.
|
|
63
92
|
#
|
|
@@ -27,6 +27,7 @@ module Olaf
|
|
|
27
27
|
|
|
28
28
|
raise UndefinedArgumentsError, self if undefined_arguments.any?
|
|
29
29
|
raise MissingArgumentsError, self if missing_arguments.any?
|
|
30
|
+
raise UnknownArgumentsError, self if unknown_arguments.any?
|
|
30
31
|
|
|
31
32
|
literal_arguments = self.class.arguments.select { |_k, v| v[:literal] }.keys
|
|
32
33
|
|
|
@@ -57,15 +58,21 @@ module Olaf
|
|
|
57
58
|
defined_arguments - @variables.keys
|
|
58
59
|
end
|
|
59
60
|
|
|
60
|
-
# Find placeholders in the SQL file.
|
|
61
61
|
# Every placeholder MUST be declared as an `argument` of the query.
|
|
62
62
|
def undefined_arguments
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
63
|
+
placeholders - defined_arguments
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
# Every `argument` declared in the query MUST have a placeholder in the SQL file.
|
|
67
|
+
def unknown_arguments
|
|
68
|
+
defined_arguments - placeholders
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
private
|
|
72
|
+
|
|
73
|
+
# Find placeholders in the SQL file.
|
|
74
|
+
def placeholders
|
|
75
|
+
@placeholders ||= @sql_template.scan(/[^:]:(\w+)/).flatten.map(&:to_sym).uniq
|
|
69
76
|
end
|
|
70
77
|
end
|
|
71
78
|
end
|
data/lib/olaf.rb
CHANGED
|
@@ -1,46 +1,83 @@
|
|
|
1
|
-
require 'sequel'
|
|
2
|
-
require 'odbc_utf8'
|
|
3
|
-
|
|
4
1
|
require_relative 'olaf/errors'
|
|
5
2
|
require_relative 'olaf/query_definition'
|
|
6
|
-
require_relative 'olaf/drivers/fake'
|
|
7
|
-
require_relative 'olaf/drivers/snowflake'
|
|
8
3
|
|
|
9
4
|
module Olaf
|
|
10
|
-
#
|
|
11
|
-
|
|
12
|
-
|
|
5
|
+
# Autoloaded: each driver pulls in its own client library.
|
|
6
|
+
autoload :BigQuery, File.expand_path('olaf/drivers/big_query', __dir__)
|
|
7
|
+
autoload :Fake, File.expand_path('olaf/drivers/fake', __dir__)
|
|
8
|
+
autoload :Snowflake, File.expand_path('olaf/drivers/snowflake', __dir__)
|
|
9
|
+
|
|
10
|
+
DEFAULT_DRIVER = :default
|
|
11
|
+
|
|
12
|
+
# Configures the drivers used to execute queries, either a single one built
|
|
13
|
+
# from the arguments given:
|
|
14
|
+
#
|
|
15
|
+
# Olaf.configure(user: 'olaf') # Olaf::Snowflake, the default
|
|
16
|
+
# Olaf.configure(olaf_driver: Olaf::Fake) # ideal for testing
|
|
13
17
|
#
|
|
14
|
-
#
|
|
15
|
-
|
|
16
|
-
|
|
18
|
+
# or several instantiated ones, which queries pick by name with `driver`:
|
|
19
|
+
#
|
|
20
|
+
# Olaf.configure(
|
|
21
|
+
# drivers: { snowflake: Olaf::Snowflake.new(user: 'olaf'), big_query: Olaf::BigQuery.new(project: 'carwow') },
|
|
22
|
+
# default: :snowflake
|
|
23
|
+
# )
|
|
24
|
+
#
|
|
25
|
+
# @return the default Olaf driver instance
|
|
26
|
+
def self.configure(olaf_driver: nil, drivers: nil, default: nil, **args)
|
|
27
|
+
@drivers, @default_driver =
|
|
28
|
+
if drivers
|
|
29
|
+
raise ArgumentError, 'Pass `drivers:` or a single `olaf_driver:`, not both' if olaf_driver || args.any?
|
|
30
|
+
raise ArgumentError, 'No drivers given to configure' if drivers.empty?
|
|
31
|
+
|
|
32
|
+
default_driver = default || drivers.keys.first
|
|
33
|
+
unless drivers.key?(default_driver)
|
|
34
|
+
raise ArgumentError, "Unknown default driver #{default_driver.inspect}, given: #{drivers.keys.inspect}"
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
[drivers.dup, default_driver]
|
|
38
|
+
else
|
|
39
|
+
[{ DEFAULT_DRIVER => (olaf_driver || Snowflake).new(**args) }, DEFAULT_DRIVER]
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
instance
|
|
17
43
|
end
|
|
18
44
|
|
|
19
|
-
# Executes a query defined by Olaf::QueryDefinition with the driver
|
|
20
|
-
#
|
|
45
|
+
# Executes a query defined by Olaf::QueryDefinition with the driver the query
|
|
46
|
+
# declares, or with the default driver when it declares none.
|
|
21
47
|
#
|
|
22
48
|
# @return Enumerable of results.
|
|
23
49
|
# (i.e. Array of Hashes or `row_objects` when specified)
|
|
24
50
|
#
|
|
25
51
|
# @raises Olaf::QueryExecutionError
|
|
52
|
+
# @raises Olaf::UnknownDriverError
|
|
26
53
|
def self.execute(olaf_query)
|
|
27
54
|
row_object = olaf_query.class.row_object
|
|
28
55
|
row_transformer = row_object ? ->(r) { row_object.new(**r) } : Proc.new(&:itself)
|
|
29
56
|
|
|
30
|
-
instance
|
|
57
|
+
instance(driver_name(olaf_query))
|
|
31
58
|
.fetch(olaf_query)
|
|
32
59
|
.map!(&row_transformer)
|
|
33
|
-
rescue Sequel::DatabaseError => error
|
|
34
|
-
raise QueryExecutionError.new(error.message, olaf_query)
|
|
35
60
|
end
|
|
36
61
|
|
|
37
|
-
# Returns an instance to execute queries when its configured.
|
|
62
|
+
# Returns an instance to execute queries when its configured. Without a name,
|
|
63
|
+
# the default driver. A single configured driver serves every name, which is
|
|
64
|
+
# what keeps `Olaf::Fake` covering every query in tests.
|
|
38
65
|
#
|
|
39
66
|
# @return Olaf driver instance
|
|
40
67
|
# * Olaf::Fake - Ideal for testing
|
|
41
68
|
# * Olaf::Snowflake - Sequel.odbc driver to run queries in Snowflake
|
|
69
|
+
# * Olaf::BigQuery - google-cloud-bigquery driver to run queries in BigQuery
|
|
42
70
|
#
|
|
43
|
-
def self.instance
|
|
44
|
-
@
|
|
71
|
+
def self.instance(name = nil)
|
|
72
|
+
drivers = @drivers || raise('You need to configure Olaf before using it!')
|
|
73
|
+
|
|
74
|
+
return drivers.fetch(@default_driver) if name.nil? || drivers.size == 1
|
|
75
|
+
|
|
76
|
+
drivers.fetch(name) { raise UnknownDriverError.new(name, drivers.keys) }
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
def self.driver_name(olaf_query)
|
|
80
|
+
olaf_query.class.driver if olaf_query.class.respond_to?(:driver)
|
|
45
81
|
end
|
|
82
|
+
private_class_method :driver_name
|
|
46
83
|
end
|
data/olaf.gemspec
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
Gem::Specification.new do |s|
|
|
2
2
|
s.name = 'olaf'
|
|
3
|
-
s.version = '0.
|
|
3
|
+
s.version = '0.3.0'
|
|
4
4
|
s.date = Time.now.strftime('%Y-%m-%d')
|
|
5
|
-
s.summary = 'Ruby wrapper for
|
|
5
|
+
s.summary = 'Ruby wrapper for warehouse queries.'
|
|
6
6
|
s.authors = ['Emiliano Mancuso']
|
|
7
7
|
s.email = ['emiliano.mancuso@gmail.com', 'developers@carwow.co.uk']
|
|
8
8
|
s.homepage = 'http://github.com/carwow/olaf'
|
|
@@ -16,7 +16,11 @@ Gem::Specification.new do |s|
|
|
|
16
16
|
]
|
|
17
17
|
s.test_files = Dir['test/*.*']
|
|
18
18
|
|
|
19
|
-
|
|
20
|
-
|
|
19
|
+
# Drivers load their own client libraries, so consumers only install the ones
|
|
20
|
+
# they configure:
|
|
21
|
+
# Olaf::Snowflake - sequel and ruby-odbc
|
|
22
|
+
# Olaf::BigQuery - google-cloud-bigquery
|
|
23
|
+
s.add_development_dependency 'google-cloud-bigquery', '~> 1.64'
|
|
24
|
+
s.add_development_dependency 'sequel', '~> 5.37'
|
|
21
25
|
s.add_development_dependency 'test-unit', '~> 3.3'
|
|
22
26
|
end
|
data/test/olaf_configure_test.rb
CHANGED
|
@@ -25,4 +25,70 @@ class OlafConfigureTest < Test::Unit::TestCase
|
|
|
25
25
|
|
|
26
26
|
assert Olaf.instance.is_a?(Olaf::Snowflake)
|
|
27
27
|
end
|
|
28
|
+
|
|
29
|
+
def test_configure_returns_the_default_driver
|
|
30
|
+
assert_equal Olaf.configure(olaf_driver: Olaf::Fake), Olaf.instance
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
def test_configure_registers_several_drivers
|
|
34
|
+
snowflake = MockDriver.new
|
|
35
|
+
big_query = MockDriver.new
|
|
36
|
+
|
|
37
|
+
Olaf.configure(drivers: { snowflake: snowflake, big_query: big_query }, default: :snowflake)
|
|
38
|
+
|
|
39
|
+
assert_equal Olaf.instance, snowflake
|
|
40
|
+
assert_equal Olaf.instance(:snowflake), snowflake
|
|
41
|
+
assert_equal Olaf.instance(:big_query), big_query
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
def test_configure_defaults_to_the_first_driver_registered
|
|
45
|
+
snowflake = MockDriver.new
|
|
46
|
+
|
|
47
|
+
Olaf.configure(drivers: { snowflake: snowflake, big_query: MockDriver.new })
|
|
48
|
+
|
|
49
|
+
assert_equal Olaf.instance, snowflake
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
def test_configure_rejects_a_default_that_was_not_registered
|
|
53
|
+
assert_raise_message(/Unknown default driver :redshift/) do
|
|
54
|
+
Olaf.configure(drivers: { snowflake: MockDriver.new }, default: :redshift)
|
|
55
|
+
end
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
def test_configure_rejects_both_forms_at_once
|
|
59
|
+
assert_raise ArgumentError do
|
|
60
|
+
Olaf.configure(olaf_driver: MockDriver, drivers: { snowflake: MockDriver.new })
|
|
61
|
+
end
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
def test_configure_rejects_an_empty_list_of_drivers
|
|
65
|
+
assert_raise ArgumentError do
|
|
66
|
+
Olaf.configure(drivers: {})
|
|
67
|
+
end
|
|
68
|
+
end
|
|
69
|
+
|
|
70
|
+
def test_instance_serves_every_name_when_a_single_driver_is_configured
|
|
71
|
+
Olaf.configure(olaf_driver: Olaf::Fake)
|
|
72
|
+
|
|
73
|
+
assert_equal Olaf.instance(:big_query), Olaf.instance
|
|
74
|
+
end
|
|
75
|
+
|
|
76
|
+
def test_instance_raises_for_a_driver_that_was_not_registered
|
|
77
|
+
Olaf.configure(drivers: { snowflake: MockDriver.new, big_query: MockDriver.new })
|
|
78
|
+
|
|
79
|
+
assert_raise Olaf::UnknownDriverError do
|
|
80
|
+
Olaf.instance(:redshift)
|
|
81
|
+
end
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
def test_instance_raises_until_olaf_is_configured
|
|
85
|
+
configured_drivers = Olaf.instance_variable_get(:@drivers)
|
|
86
|
+
Olaf.instance_variable_set(:@drivers, nil)
|
|
87
|
+
|
|
88
|
+
assert_raise_message(/You need to configure Olaf before using it!/) do
|
|
89
|
+
Olaf.instance
|
|
90
|
+
end
|
|
91
|
+
ensure
|
|
92
|
+
Olaf.instance_variable_set(:@drivers, configured_drivers)
|
|
93
|
+
end
|
|
28
94
|
end
|
data/test/olaf_execute_test.rb
CHANGED
|
@@ -1,6 +1,23 @@
|
|
|
1
1
|
require_relative 'helper'
|
|
2
2
|
|
|
3
3
|
class OlafExecuteTest < Test::Unit::TestCase
|
|
4
|
+
Company = Struct.new(:company, keyword_init: true)
|
|
5
|
+
|
|
6
|
+
class RecordingDriver
|
|
7
|
+
attr_reader :executed
|
|
8
|
+
|
|
9
|
+
def initialize(**config)
|
|
10
|
+
@config = config
|
|
11
|
+
@executed = []
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
def fetch(olaf_query)
|
|
15
|
+
@executed << olaf_query
|
|
16
|
+
|
|
17
|
+
[{ company: 'carwow' }]
|
|
18
|
+
end
|
|
19
|
+
end
|
|
20
|
+
|
|
4
21
|
def setup
|
|
5
22
|
Olaf.configure(olaf_driver: Olaf::Fake)
|
|
6
23
|
|
|
@@ -17,9 +34,9 @@ class OlafExecuteTest < Test::Unit::TestCase
|
|
|
17
34
|
end
|
|
18
35
|
|
|
19
36
|
def test_execute_returns_a_list_of_row_objects_when_defined
|
|
20
|
-
@query.row_object
|
|
37
|
+
@query.row_object Company
|
|
21
38
|
|
|
22
|
-
all_row_objects = Olaf.execute(@query_instance).all? { |e| e.is_a?
|
|
39
|
+
all_row_objects = Olaf.execute(@query_instance).all? { |e| e.is_a? Company }
|
|
23
40
|
|
|
24
41
|
assert all_row_objects
|
|
25
42
|
end
|
|
@@ -30,12 +47,12 @@ class OlafExecuteTest < Test::Unit::TestCase
|
|
|
30
47
|
assert all_hashes
|
|
31
48
|
end
|
|
32
49
|
|
|
33
|
-
def
|
|
50
|
+
def test_execute_lets_the_error_wrapped_by_the_driver_through
|
|
34
51
|
faulty_driver = Class.new do
|
|
35
52
|
def initialize(**args); end
|
|
36
53
|
|
|
37
|
-
def fetch(
|
|
38
|
-
raise
|
|
54
|
+
def fetch(olaf_query)
|
|
55
|
+
raise Olaf::QueryExecutionError.new('something went wrong', olaf_query)
|
|
39
56
|
end
|
|
40
57
|
end
|
|
41
58
|
|
|
@@ -45,4 +62,48 @@ class OlafExecuteTest < Test::Unit::TestCase
|
|
|
45
62
|
Olaf.execute(@query_instance)
|
|
46
63
|
end
|
|
47
64
|
end
|
|
65
|
+
|
|
66
|
+
def test_execute_uses_the_driver_declared_by_the_query
|
|
67
|
+
default_driver = RecordingDriver.new
|
|
68
|
+
other_driver = RecordingDriver.new
|
|
69
|
+
|
|
70
|
+
Olaf.configure(drivers: { default: default_driver, big_query: other_driver })
|
|
71
|
+
|
|
72
|
+
@query.driver :big_query
|
|
73
|
+
|
|
74
|
+
Olaf.execute(@query_instance)
|
|
75
|
+
|
|
76
|
+
assert_equal other_driver.executed, [@query_instance]
|
|
77
|
+
assert_equal default_driver.executed, []
|
|
78
|
+
end
|
|
79
|
+
|
|
80
|
+
def test_execute_uses_the_default_driver_when_the_query_declares_none
|
|
81
|
+
default_driver = RecordingDriver.new
|
|
82
|
+
other_driver = RecordingDriver.new
|
|
83
|
+
|
|
84
|
+
Olaf.configure(drivers: { big_query: other_driver, snowflake: default_driver }, default: :snowflake)
|
|
85
|
+
|
|
86
|
+
Olaf.execute(@query_instance)
|
|
87
|
+
|
|
88
|
+
assert_equal default_driver.executed, [@query_instance]
|
|
89
|
+
assert_equal other_driver.executed, []
|
|
90
|
+
end
|
|
91
|
+
|
|
92
|
+
def test_execute_uses_the_only_driver_configured_whatever_the_query_declares
|
|
93
|
+
@query.driver :big_query
|
|
94
|
+
|
|
95
|
+
Olaf.execute(@query_instance)
|
|
96
|
+
|
|
97
|
+
assert_equal Olaf.instance.execution_log, [@query_instance]
|
|
98
|
+
end
|
|
99
|
+
|
|
100
|
+
def test_execute_raises_when_the_driver_declared_is_not_configured
|
|
101
|
+
Olaf.configure(drivers: { default: RecordingDriver.new, snowflake: RecordingDriver.new })
|
|
102
|
+
|
|
103
|
+
@query.driver :big_query
|
|
104
|
+
|
|
105
|
+
assert_raise_message(/Unknown driver :big_query/) do
|
|
106
|
+
Olaf.execute(@query_instance)
|
|
107
|
+
end
|
|
108
|
+
end
|
|
48
109
|
end
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
require_relative 'helper'
|
|
2
|
+
|
|
3
|
+
# Drivers pull in client libraries that not every consumer installs, so what
|
|
4
|
+
# gets loaded, and when, is part of the interface. These run in a subprocess:
|
|
5
|
+
# the rest of the suite has all of them loaded already.
|
|
6
|
+
class OlafLazyLoadingTest < Test::Unit::TestCase
|
|
7
|
+
def test_requiring_olaf_loads_no_driver_dependency
|
|
8
|
+
assert_equal loaded_after('require "olaf"'), []
|
|
9
|
+
end
|
|
10
|
+
|
|
11
|
+
def test_configuring_the_fake_driver_loads_no_driver_dependency
|
|
12
|
+
assert_equal loaded_after('require "olaf"; Olaf.configure(olaf_driver: Olaf::Fake)'), []
|
|
13
|
+
end
|
|
14
|
+
|
|
15
|
+
def test_configuring_the_snowflake_driver_loads_sequel_only
|
|
16
|
+
assert_equal loaded_after('require "olaf"; Olaf.configure(user: "olaf")'), ['sequel']
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
private
|
|
20
|
+
|
|
21
|
+
DEPENDENCIES = {
|
|
22
|
+
'sequel' => 'Sequel',
|
|
23
|
+
'odbc_utf8' => 'ODBC',
|
|
24
|
+
'google-cloud-bigquery' => 'Google::Cloud::Bigquery'
|
|
25
|
+
}.freeze
|
|
26
|
+
|
|
27
|
+
def loaded_after(script)
|
|
28
|
+
report = "print #{DEPENDENCIES.values.inspect}.map { |c| Object.const_defined?(c) }.inspect"
|
|
29
|
+
# RUBYOPT carries bundler into the subprocess, which is not what is measured.
|
|
30
|
+
output = IO.popen({ 'RUBYOPT' => nil }, [RbConfig.ruby, '-I', lib_path, '-e', "#{script}; #{report}"], &:read)
|
|
31
|
+
|
|
32
|
+
assert $?.success?, "Failed to run: #{script}"
|
|
33
|
+
|
|
34
|
+
DEPENDENCIES.keys.zip(eval(output)).select { |_name, loaded| loaded }.map(&:first)
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
def lib_path
|
|
38
|
+
File.expand_path('../lib', File.dirname(__FILE__))
|
|
39
|
+
end
|
|
40
|
+
end
|
metadata
CHANGED
|
@@ -1,43 +1,42 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: olaf
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.3.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Emiliano Mancuso
|
|
8
|
-
autorequire:
|
|
9
8
|
bindir: bin
|
|
10
9
|
cert_chain: []
|
|
11
|
-
date:
|
|
10
|
+
date: 2026-09-03 00:00:00.000000000 Z
|
|
12
11
|
dependencies:
|
|
13
12
|
- !ruby/object:Gem::Dependency
|
|
14
|
-
name:
|
|
13
|
+
name: google-cloud-bigquery
|
|
15
14
|
requirement: !ruby/object:Gem::Requirement
|
|
16
15
|
requirements:
|
|
17
16
|
- - "~>"
|
|
18
17
|
- !ruby/object:Gem::Version
|
|
19
|
-
version: '
|
|
20
|
-
type: :
|
|
18
|
+
version: '1.64'
|
|
19
|
+
type: :development
|
|
21
20
|
prerelease: false
|
|
22
21
|
version_requirements: !ruby/object:Gem::Requirement
|
|
23
22
|
requirements:
|
|
24
23
|
- - "~>"
|
|
25
24
|
- !ruby/object:Gem::Version
|
|
26
|
-
version: '
|
|
25
|
+
version: '1.64'
|
|
27
26
|
- !ruby/object:Gem::Dependency
|
|
28
|
-
name:
|
|
27
|
+
name: sequel
|
|
29
28
|
requirement: !ruby/object:Gem::Requirement
|
|
30
29
|
requirements:
|
|
31
30
|
- - "~>"
|
|
32
31
|
- !ruby/object:Gem::Version
|
|
33
|
-
version: '
|
|
34
|
-
type: :
|
|
32
|
+
version: '5.37'
|
|
33
|
+
type: :development
|
|
35
34
|
prerelease: false
|
|
36
35
|
version_requirements: !ruby/object:Gem::Requirement
|
|
37
36
|
requirements:
|
|
38
37
|
- - "~>"
|
|
39
38
|
- !ruby/object:Gem::Version
|
|
40
|
-
version: '
|
|
39
|
+
version: '5.37'
|
|
41
40
|
- !ruby/object:Gem::Dependency
|
|
42
41
|
name: test-unit
|
|
43
42
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -52,7 +51,6 @@ dependencies:
|
|
|
52
51
|
- - "~>"
|
|
53
52
|
- !ruby/object:Gem::Version
|
|
54
53
|
version: '3.3'
|
|
55
|
-
description:
|
|
56
54
|
email:
|
|
57
55
|
- emiliano.mancuso@gmail.com
|
|
58
56
|
- developers@carwow.co.uk
|
|
@@ -62,6 +60,7 @@ extra_rdoc_files: []
|
|
|
62
60
|
files:
|
|
63
61
|
- README.md
|
|
64
62
|
- lib/olaf.rb
|
|
63
|
+
- lib/olaf/drivers/big_query.rb
|
|
65
64
|
- lib/olaf/drivers/fake.rb
|
|
66
65
|
- lib/olaf/drivers/snowflake.rb
|
|
67
66
|
- lib/olaf/errors.rb
|
|
@@ -72,11 +71,11 @@ files:
|
|
|
72
71
|
- test/helper.rb
|
|
73
72
|
- test/olaf_configure_test.rb
|
|
74
73
|
- test/olaf_execute_test.rb
|
|
74
|
+
- test/olaf_lazy_loading_test.rb
|
|
75
75
|
homepage: http://github.com/carwow/olaf
|
|
76
76
|
licenses:
|
|
77
77
|
- MIT
|
|
78
78
|
metadata: {}
|
|
79
|
-
post_install_message:
|
|
80
79
|
rdoc_options: []
|
|
81
80
|
require_paths:
|
|
82
81
|
- lib
|
|
@@ -91,11 +90,11 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
91
90
|
- !ruby/object:Gem::Version
|
|
92
91
|
version: '0'
|
|
93
92
|
requirements: []
|
|
94
|
-
rubygems_version:
|
|
95
|
-
signing_key:
|
|
93
|
+
rubygems_version: 4.0.20
|
|
96
94
|
specification_version: 4
|
|
97
|
-
summary: Ruby wrapper for
|
|
95
|
+
summary: Ruby wrapper for warehouse queries.
|
|
98
96
|
test_files:
|
|
99
97
|
- test/helper.rb
|
|
100
|
-
- test/olaf_execute_test.rb
|
|
101
98
|
- test/olaf_configure_test.rb
|
|
99
|
+
- test/olaf_execute_test.rb
|
|
100
|
+
- test/olaf_lazy_loading_test.rb
|