dwh 0.1.0 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +5 -4
- data/Rakefile +1 -1
- data/docs/DWH/Adapters/Adapter.html +33 -27
- data/docs/DWH/Adapters/Athena.html +25 -21
- data/docs/DWH/Adapters/Boolean.html +1 -1
- data/docs/DWH/Adapters/Druid.html +18 -12
- data/docs/DWH/Adapters/DuckDb.html +29 -27
- data/docs/DWH/Adapters/MySql.html +25 -19
- data/docs/DWH/Adapters/OpenAuthorizable/ClassMethods.html +3 -6
- data/docs/DWH/Adapters/OpenAuthorizable.html +5 -10
- data/docs/DWH/Adapters/Postgres.html +27 -23
- data/docs/DWH/Adapters/Snowflake.html +39 -24
- data/docs/DWH/Adapters/SqlServer.html +27 -25
- data/docs/DWH/Adapters/Trino.html +30 -30
- data/docs/DWH/Adapters.html +1 -1
- data/docs/DWH/AuthenticationError.html +1 -1
- data/docs/DWH/Behaviors.html +6 -11
- data/docs/DWH/Capabilities.html +10 -26
- data/docs/DWH/Column.html +7 -15
- data/docs/DWH/ConfigError.html +1 -1
- data/docs/DWH/ConnectionError.html +1 -1
- data/docs/DWH/DWHError.html +1 -1
- data/docs/DWH/ExecutionError.html +1 -1
- data/docs/DWH/Factory.html +1 -1
- data/docs/DWH/Functions/Arrays.html +8 -8
- data/docs/DWH/Functions/Dates.html +5 -7
- data/docs/DWH/Functions/ExtractDatePart.html +13 -25
- data/docs/DWH/Functions/Nulls.html +3 -3
- data/docs/DWH/Functions.html +6 -9
- data/docs/DWH/Logger.html +3 -5
- data/docs/DWH/OAuthError.html +1 -1
- data/docs/DWH/Settings.html +6 -9
- data/docs/DWH/StreamingStats.html +2 -3
- data/docs/DWH/Table.html +14 -26
- data/docs/DWH/TableStats.html +1 -1
- data/docs/DWH/TokenExpiredError.html +1 -1
- data/docs/DWH/UnsupportedCapability.html +1 -1
- data/docs/DWH.html +1 -1
- data/docs/_index.html +1 -1
- data/docs/file.README.html +43 -48
- data/docs/file.adapters.html +318 -343
- data/docs/file.creating-adapters.html +347 -357
- data/docs/file.getting-started.html +143 -151
- data/docs/file.usage.html +257 -278
- data/docs/index.html +43 -48
- data/docs/top-level-namespace.html +1 -1
- data/lib/dwh/adapters.rb +2 -2
- data/lib/dwh/version.rb +1 -1
- metadata +2 -2
@@ -63,222 +63,214 @@
|
|
63
63
|
|
64
64
|
<h2 id="installation">Installation</h2>
|
65
65
|
|
66
|
-
<p>Add this line to your application
|
66
|
+
<p>Add this line to your application's Gemfile:</p>
|
67
67
|
|
68
|
-
<
|
69
|
-
|
70
|
-
</code></p>
|
68
|
+
<pre class="code ruby"><code class="ruby"><span class='id identifier rubyid_gem'>gem</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>dwh</span><span class='tstring_end'>'</span></span>
|
69
|
+
</code></pre>
|
71
70
|
|
72
71
|
<p>And then execute:</p>
|
73
72
|
|
74
|
-
<
|
75
|
-
|
76
|
-
</code></p>
|
73
|
+
<pre class="code bash"><code class="bash">bundle install
|
74
|
+
</code></pre>
|
77
75
|
|
78
76
|
<p>Or install it yourself as:</p>
|
79
77
|
|
80
|
-
<
|
81
|
-
|
82
|
-
</code></p>
|
78
|
+
<pre class="code bash"><code class="bash">gem install dwh
|
79
|
+
</code></pre>
|
83
80
|
|
84
81
|
<h2 id="basic-usage">Basic Usage</h2>
|
85
82
|
|
86
83
|
<h3 id="creating-your-first-connection">Creating Your First Connection</h3>
|
87
84
|
|
88
|
-
<
|
89
|
-
require ‘dwh’</p>
|
85
|
+
<pre class="code ruby"><code class="ruby"><span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>dwh</span><span class='tstring_end'>'</span></span>
|
90
86
|
|
91
|
-
<
|
92
|
-
|
93
|
-
host
|
94
|
-
database
|
95
|
-
username
|
96
|
-
password
|
97
|
-
})</
|
87
|
+
<span class='comment'># Connect to PostgreSQL
|
88
|
+
</span><span class='id identifier rubyid_postgres'>postgres</span> <span class='op'>=</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='period'>.</span><span class='id identifier rubyid_create'><span class='object_link'><a href="DWH/Factory.html#create-instance_method" title="DWH::Factory#create (method)">create</a></span></span><span class='lparen'>(</span><span class='symbol'>:postgres</span><span class='comma'>,</span> <span class='lbrace'>{</span>
|
89
|
+
<span class='label'>host:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>localhost</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
90
|
+
<span class='label'>database:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>mydb</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
91
|
+
<span class='label'>username:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>user</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
92
|
+
<span class='label'>password:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>password</span><span class='tstring_end'>'</span></span>
|
93
|
+
<span class='rbrace'>}</span><span class='rparen'>)</span>
|
98
94
|
|
99
|
-
<
|
100
|
-
|
101
|
-
database
|
102
|
-
})
|
103
|
-
|
95
|
+
<span class='comment'># Connect to DuckDB (in-memory)
|
96
|
+
</span><span class='id identifier rubyid_duckdb'>duckdb</span> <span class='op'>=</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='period'>.</span><span class='id identifier rubyid_create'><span class='object_link'><a href="DWH/Factory.html#create-instance_method" title="DWH::Factory#create (method)">create</a></span></span><span class='lparen'>(</span><span class='symbol'>:duckdb</span><span class='comma'>,</span> <span class='lbrace'>{</span>
|
97
|
+
<span class='label'>database:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>:memory:</span><span class='tstring_end'>'</span></span>
|
98
|
+
<span class='rbrace'>}</span><span class='rparen'>)</span>
|
99
|
+
</code></pre>
|
104
100
|
|
105
101
|
<h3 id="your-first-query">Your First Query</h3>
|
106
102
|
|
107
|
-
<
|
108
|
-
|
109
|
-
results = postgres.execute(“SELECT * FROM users LIMIT 10”)</p>
|
103
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Execute a simple query
|
104
|
+
</span><span class='id identifier rubyid_results'>results</span> <span class='op'>=</span> <span class='id identifier rubyid_postgres'>postgres</span><span class='period'>.</span><span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SELECT * FROM users LIMIT 10</span><span class='tstring_end'>"</span></span><span class='rparen'>)</span>
|
110
105
|
|
111
|
-
<
|
112
|
-
|
113
|
-
puts row
|
114
|
-
end
|
115
|
-
|
106
|
+
<span class='comment'># Results are returned as arrays by default
|
107
|
+
</span><span class='id identifier rubyid_results'>results</span><span class='period'>.</span><span class='id identifier rubyid_each'>each</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_row'>row</span><span class='op'>|</span>
|
108
|
+
<span class='id identifier rubyid_puts'>puts</span> <span class='id identifier rubyid_row'>row</span><span class='period'>.</span><span class='id identifier rubyid_inspect'>inspect</span>
|
109
|
+
<span class='kw'>end</span>
|
110
|
+
</code></pre>
|
116
111
|
|
117
112
|
<h3 id="exploring-your-database">Exploring Your Database</h3>
|
118
113
|
|
119
|
-
<
|
120
|
-
|
121
|
-
tables =
|
122
|
-
|
123
|
-
|
124
|
-
<
|
125
|
-
<
|
126
|
-
puts
|
127
|
-
puts
|
128
|
-
|
129
|
-
|
130
|
-
|
131
|
-
|
132
|
-
|
133
|
-
<
|
134
|
-
<
|
135
|
-
puts
|
136
|
-
|
137
|
-
```</p>
|
114
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># List all tables
|
115
|
+
</span><span class='id identifier rubyid_tables'>tables</span> <span class='op'>=</span> <span class='id identifier rubyid_postgres'>postgres</span><span class='period'>.</span><span class='id identifier rubyid_tables'>tables</span>
|
116
|
+
<span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Available tables: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_tables'>tables</span><span class='period'>.</span><span class='id identifier rubyid_map'>map</span><span class='lparen'>(</span><span class='op'>&</span><span class='symbol'>:physical_name</span><span class='rparen'>)</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
117
|
+
|
118
|
+
<span class='comment'># Get detailed information about a table
|
119
|
+
</span><span class='id identifier rubyid_table_info'>table_info</span> <span class='op'>=</span> <span class='id identifier rubyid_postgres'>postgres</span><span class='period'>.</span><span class='id identifier rubyid_metadata'>metadata</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>users</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span>
|
120
|
+
<span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Table: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_table_info'>table_info</span><span class='period'>.</span><span class='id identifier rubyid_physical_name'>physical_name</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
121
|
+
<span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Schema: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_table_info'>table_info</span><span class='period'>.</span><span class='id identifier rubyid_schema'>schema</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
122
|
+
<span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Columns:</span><span class='tstring_end'>"</span></span>
|
123
|
+
<span class='id identifier rubyid_table_info'>table_info</span><span class='period'>.</span><span class='id identifier rubyid_columns'>columns</span><span class='period'>.</span><span class='id identifier rubyid_each'>each</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_column'>column</span><span class='op'>|</span>
|
124
|
+
<span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'> </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_column'>column</span><span class='period'>.</span><span class='id identifier rubyid_name'>name</span><span class='embexpr_end'>}</span><span class='tstring_content'> (</span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_column'>column</span><span class='period'>.</span><span class='id identifier rubyid_normalized_data_type'>normalized_data_type</span><span class='embexpr_end'>}</span><span class='tstring_content'>)</span><span class='tstring_end'>"</span></span>
|
125
|
+
<span class='kw'>end</span>
|
126
|
+
|
127
|
+
<span class='comment'># Get table statistics
|
128
|
+
</span><span class='id identifier rubyid_stats'>stats</span> <span class='op'>=</span> <span class='id identifier rubyid_postgres'>postgres</span><span class='period'>.</span><span class='id identifier rubyid_stats'>stats</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>users</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='label'>date_column:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>created_at</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span>
|
129
|
+
<span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Row count: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_stats'>stats</span><span class='period'>.</span><span class='id identifier rubyid_row_count'>row_count</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
130
|
+
<span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Date range: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_stats'>stats</span><span class='period'>.</span><span class='id identifier rubyid_date_start'>date_start</span><span class='embexpr_end'>}</span><span class='tstring_content'> to </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_stats'>stats</span><span class='period'>.</span><span class='id identifier rubyid_date_end'>date_end</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
131
|
+
</code></pre>
|
138
132
|
|
139
133
|
<h3 id="different-output-formats">Different Output Formats</h3>
|
140
134
|
|
141
|
-
<
|
142
|
-
|
143
|
-
array_results = postgres.execute(“SELECT id, name FROM users LIMIT 5”)</p>
|
135
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Get results as arrays (default)
|
136
|
+
</span><span class='id identifier rubyid_array_results'>array_results</span> <span class='op'>=</span> <span class='id identifier rubyid_postgres'>postgres</span><span class='period'>.</span><span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SELECT id, name FROM users LIMIT 5</span><span class='tstring_end'>"</span></span><span class='rparen'>)</span>
|
144
137
|
|
145
|
-
<
|
146
|
-
|
138
|
+
<span class='comment'># Get results as hashes/objects
|
139
|
+
</span><span class='id identifier rubyid_hash_results'>hash_results</span> <span class='op'>=</span> <span class='id identifier rubyid_postgres'>postgres</span><span class='period'>.</span><span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SELECT id, name FROM users LIMIT 5</span><span class='tstring_end'>"</span></span><span class='comma'>,</span> <span class='label'>format:</span> <span class='symbol'>:object</span><span class='rparen'>)</span>
|
147
140
|
|
148
|
-
<
|
149
|
-
|
141
|
+
<span class='comment'># Get results as CSV string
|
142
|
+
</span><span class='id identifier rubyid_csv_results'>csv_results</span> <span class='op'>=</span> <span class='id identifier rubyid_postgres'>postgres</span><span class='period'>.</span><span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SELECT id, name FROM users LIMIT 5</span><span class='tstring_end'>"</span></span><span class='comma'>,</span> <span class='label'>format:</span> <span class='symbol'>:csv</span><span class='rparen'>)</span>
|
150
143
|
|
151
|
-
<
|
152
|
-
|
153
|
-
|
144
|
+
<span class='comment'># Stream large results to a file
|
145
|
+
</span><span class='id identifier rubyid_postgres'>postgres</span><span class='period'>.</span><span class='id identifier rubyid_execute_stream'>execute_stream</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SELECT * FROM large_table</span><span class='tstring_end'>"</span></span><span class='comma'>,</span> <span class='const'>File</span><span class='period'>.</span><span class='id identifier rubyid_open'>open</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>output.csv</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>w</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span><span class='rparen'>)</span>
|
146
|
+
</code></pre>
|
154
147
|
|
155
148
|
<h3 id="streaming-large-datasets">Streaming Large Datasets</h3>
|
156
149
|
|
157
|
-
<
|
158
|
-
#
|
159
|
-
|
160
|
-
|
161
|
-
|
162
|
-
}</p>
|
150
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># stream data while tracting stats and previewing data in a separate thread
|
151
|
+
</span><span class='id identifier rubyid_stats'>stats</span> <span class='op'>=</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="DWH/StreamingStats.html" title="DWH::StreamingStats (class)">StreamingStats</a></span></span><span class='period'>.</span><span class='id identifier rubyid_new'><span class='object_link'><a href="DWH/StreamingStats.html#initialize-instance_method" title="DWH::StreamingStats#initialize (method)">new</a></span></span><span class='lparen'>(</span><span class='int'>10000</span><span class='rparen'>)</span> <span class='comment'># num of rows to keep in memory for previewing
|
152
|
+
</span><span class='id identifier rubyid_exec_thread'>exec_thread</span> <span class='op'>=</span> <span class='const'>Thread</span><span class='period'>.</span><span class='id identifier rubyid_new'>new</span> <span class='lbrace'>{</span>
|
153
|
+
<span class='id identifier rubyid_postgres'>postgres</span><span class='period'>.</span><span class='id identifier rubyid_execute_stream'>execute_stream</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SELECT * FROM large_table</span><span class='tstring_end'>"</span></span><span class='comma'>,</span> <span class='const'>File</span><span class='period'>.</span><span class='id identifier rubyid_open'>open</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>output.csv</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>w</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span><span class='comma'>,</span> <span class='label'>stats:</span> <span class='id identifier rubyid_stats'>stats</span><span class='rparen'>)</span>
|
154
|
+
<span class='rbrace'>}</span>
|
163
155
|
|
164
|
-
<
|
165
|
-
loop do
|
166
|
-
break if exec_thread
|
156
|
+
<span class='id identifier rubyid_mon_thread'>mon_thread</span> <span class='op'>=</span> <span class='const'>Thread</span><span class='period'>.</span><span class='id identifier rubyid_new'>new</span><span class='lbrace'>{</span>
|
157
|
+
<span class='id identifier rubyid_loop'>loop</span> <span class='kw'>do</span>
|
158
|
+
<span class='kw'>break</span> <span class='kw'>if</span> <span class='id identifier rubyid_exec_thread'>exec_thread</span><span class='period'>.</span><span class='id identifier rubyid_alive?'>alive?</span>
|
167
159
|
|
168
|
-
<
|
169
|
-
</
|
160
|
+
<span class='id identifier rubyid_puts'>puts</span> <span class='id identifier rubyid_stats'>stats</span><span class='period'>.</span><span class='id identifier rubyid_data'>data</span><span class='period'>.</span><span class='id identifier rubyid_last'>last</span>
|
161
|
+
<span class='kw'>end</span>
|
162
|
+
<span class='rbrace'>}</span>
|
170
163
|
|
171
|
-
<
|
164
|
+
<span class='lbracket'>[</span><span class='id identifier rubyid_exec_thread'>exec_thread</span><span class='comma'>,</span> <span class='id identifier rubyid_mon_thread'>mon_thread</span><span class='rbracket'>]</span><span class='period'>.</span><span class='id identifier rubyid_each'>each</span><span class='lparen'>(</span><span class='op'>&</span><span class='symbol'>:join</span><span class='rparen'>)</span>
|
172
165
|
|
173
|
-
<
|
174
|
-
|
175
|
-
process_chunk(chunk)
|
176
|
-
end</
|
166
|
+
<span class='comment'># Stream with block processing
|
167
|
+
</span><span class='id identifier rubyid_postgres'>postgres</span><span class='period'>.</span><span class='id identifier rubyid_stream'>stream</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SELECT * FROM large_table</span><span class='tstring_end'>"</span></span><span class='rparen'>)</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_chunk'>chunk</span><span class='op'>|</span>
|
168
|
+
<span class='id identifier rubyid_process_chunk'>process_chunk</span><span class='lparen'>(</span><span class='id identifier rubyid_chunk'>chunk</span><span class='rparen'>)</span>
|
169
|
+
<span class='kw'>end</span>
|
177
170
|
|
178
|
-
|
171
|
+
</code></pre>
|
179
172
|
|
180
173
|
<h2 id="advanced-usage">Advanced Usage</h2>
|
181
174
|
|
182
175
|
<h3 id="connection-pooling">Connection Pooling</h3>
|
183
176
|
|
184
|
-
<
|
185
|
-
#
|
186
|
-
|
187
|
-
|
188
|
-
|
189
|
-
|
190
|
-
|
191
|
-
|
192
|
-
|
193
|
-
|
194
|
-
<
|
195
|
-
|
196
|
-
|
197
|
-
|
198
|
-
|
199
|
-
|
200
|
-
```</p>
|
177
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Create a connection pool
|
178
|
+
</span><span class='id identifier rubyid_pool'>pool</span> <span class='op'>=</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='period'>.</span><span class='id identifier rubyid_pool'><span class='object_link'><a href="DWH/Factory.html#pool-instance_method" title="DWH::Factory#pool (method)">pool</a></span></span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>my_postgres_pool</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='symbol'>:postgres</span><span class='comma'>,</span> <span class='lbrace'>{</span>
|
179
|
+
<span class='label'>host:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>localhost</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
180
|
+
<span class='label'>database:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>mydb</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
181
|
+
<span class='label'>username:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>user</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
182
|
+
<span class='label'>password:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>password</span><span class='tstring_end'>'</span></span>
|
183
|
+
<span class='rbrace'>}</span><span class='comma'>,</span> <span class='label'>size:</span> <span class='int'>10</span><span class='comma'>,</span> <span class='label'>timeout:</span> <span class='int'>5</span><span class='rparen'>)</span>
|
184
|
+
|
185
|
+
<span class='comment'># Use the pool
|
186
|
+
</span><span class='id identifier rubyid_pool'>pool</span><span class='period'>.</span><span class='id identifier rubyid_with'>with</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_connection'>connection</span><span class='op'>|</span>
|
187
|
+
<span class='id identifier rubyid_results'>results</span> <span class='op'>=</span> <span class='id identifier rubyid_connection'>connection</span><span class='period'>.</span><span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SELECT COUNT(*) FROM users</span><span class='tstring_end'>"</span></span><span class='rparen'>)</span>
|
188
|
+
<span class='kw'>end</span>
|
189
|
+
|
190
|
+
<span class='comment'># Shutdown the pool when done
|
191
|
+
</span><span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='period'>.</span><span class='id identifier rubyid_shutdown'><span class='object_link'><a href="DWH/Factory.html#shutdown-instance_method" title="DWH::Factory#shutdown (method)">shutdown</a></span></span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>my_postgres_pool</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span>
|
192
|
+
</code></pre>
|
201
193
|
|
202
194
|
<h3 id="using-extra-connection-params">Using Extra Connection Params</h3>
|
203
195
|
|
204
196
|
<p>DWH uses an existing Ruby gem where possible to connect to each target database. When that is not possible and the db supports a REST endpoint, we will use Faraday.</p>
|
205
197
|
|
206
|
-
<p>Using <code>extra_connection_params</code> key you can pass in a Hash of options that the target connector supports but DWH doesn
|
198
|
+
<p>Using <code>extra_connection_params</code> key you can pass in a Hash of options that the target connector supports but DWH doesn't make first class. The main config options in DWH are based on required and common needs.</p>
|
207
199
|
|
208
|
-
<h4 id="sending-postgres-
|
200
|
+
<h4 id="sending-postgres-connect_timeout-property-supported-by-the-pg-gem">Sending Postgres 'connect_timeout' property supported by the PG gem</h4>
|
209
201
|
|
210
|
-
<
|
211
|
-
|
212
|
-
|
213
|
-
|
214
|
-
|
215
|
-
|
216
|
-
|
217
|
-
|
218
|
-
|
219
|
-
})</p>
|
202
|
+
<pre class="code ruby"><code class="ruby"><span class='id identifier rubyid_pg'>pg</span> <span class='op'>=</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='period'>.</span><span class='id identifier rubyid_create'><span class='object_link'><a href="DWH/Factory.html#create-instance_method" title="DWH::Factory#create (method)">create</a></span></span><span class='lparen'>(</span><span class='symbol'>:postgres</span><span class='comma'>,</span> <span class='lbrace'>{</span>
|
203
|
+
<span class='label'>host:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>localhost</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
204
|
+
<span class='label'>database:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>mydb</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
205
|
+
<span class='label'>username:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>user</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
206
|
+
<span class='label'>password:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>password</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
207
|
+
<span class='label'>extra_connection_params:</span> <span class='lbrace'>{</span>
|
208
|
+
<span class='label'>connect_timeout:</span> <span class='int'>5</span>
|
209
|
+
<span class='rbrace'>}</span>
|
210
|
+
<span class='rbrace'>}</span><span class='rparen'>)</span>
|
220
211
|
|
221
|
-
|
212
|
+
</code></pre>
|
222
213
|
|
223
214
|
<h3 id="database-functions">Database Functions</h3>
|
224
215
|
|
225
216
|
<p>DWH provides a function translation layer that converts common SQL functions to database-specific syntax:</p>
|
226
217
|
|
227
|
-
<
|
228
|
-
|
229
|
-
|
230
|
-
|
218
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Date truncation
|
219
|
+
</span><span class='id identifier rubyid_postgres'>postgres</span><span class='period'>.</span><span class='id identifier rubyid_truncate_date'>truncate_date</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>week</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>created_at</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span> <span class='comment'># => DATE_TRUNC('week', created_at)
|
220
|
+
</span><span class='id identifier rubyid_sqlserver'>sqlserver</span><span class='period'>.</span><span class='id identifier rubyid_truncate_date'>truncate_date</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>week</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>created_at</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span> <span class='comment'># => DATETRUNC(week, created_at)
|
221
|
+
</span>
|
222
|
+
<span class='comment'># Date literals
|
223
|
+
</span><span class='id identifier rubyid_postgres'>postgres</span><span class='period'>.</span><span class='id identifier rubyid_date_literal'>date_literal</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>2025-01-01</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span> <span class='comment'># => '2025-01-01'::DATE
|
224
|
+
</span><span class='id identifier rubyid_sqlserver'>sqlserver</span><span class='period'>.</span><span class='id identifier rubyid_date_literal'>date_literal</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>2025-01-01</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span> <span class='comment'># => '2025-01-01'
|
225
|
+
</span>
|
226
|
+
<span class='comment'># Null handling
|
227
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_coalesce'>coalesce</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>column1</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>column2</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>'default'</span><span class='tstring_end'>"</span></span><span class='rparen'>)</span> <span class='comment'># => COALESCE(column1, column2, 'default')
|
228
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_null_if'>null_if</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>column1</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>'empty'</span><span class='tstring_end'>"</span></span><span class='rparen'>)</span> <span class='comment'># => NULLIF(column1, 'empty')
|
229
|
+
</span>
|
230
|
+
<span class='comment'># String functions
|
231
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_trim'>trim</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>column_name</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span> <span class='comment'># => TRIM(column_name)
|
232
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_upper_case'>upper_case</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>column_name</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span> <span class='comment'># => UPPER(column_name)
|
233
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_lower_case'>lower_case</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>column_name</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span> <span class='comment'># => LOWER(column_name)
|
234
|
+
</span></code></pre>
|
235
|
+
|
236
|
+
<h2 id="core-api">Core API</h2>
|
237
|
+
|
238
|
+
<p>Standardized API across adapters:</p>
|
231
239
|
|
232
|
-
<
|
233
|
-
|
234
|
-
sqlserver.date_literal(‘2025-01-01’) # => ‘2025-01-01’</p>
|
240
|
+
<p>connection
|
241
|
+
: creates a reusuable connection based on config hash passed in</p>
|
235
242
|
|
236
|
-
<
|
237
|
-
|
238
|
-
adapter.null_if(‘column1’, “‘empty’”) # => NULLIF(column1, ‘empty’)</p>
|
243
|
+
<p>tables(schema: nil, catalog: nil)
|
244
|
+
: returns a list of tables from the default connection or from the specified schema and catalog</p>
|
239
245
|
|
240
|
-
<
|
241
|
-
|
242
|
-
adapter.upper_case(‘column_name’) # => UPPER(column_name)
|
243
|
-
adapter.lower_case(‘column_name’) # => LOWER(column_name)
|
244
|
-
```</p>
|
246
|
+
<p>metadata(table_name, schema: nil, catalog: nil)
|
247
|
+
: provides metadata about a table</p>
|
245
248
|
|
246
|
-
<
|
249
|
+
<p>stats(table_name, date_column: nil)
|
250
|
+
: provides table row count and date range</p>
|
247
251
|
|
248
|
-
<p>
|
252
|
+
<p>execute(sql, format: :array, retries: 0)
|
253
|
+
: runs a query and returns in given format</p>
|
249
254
|
|
250
|
-
<
|
251
|
-
|
252
|
-
<dd>creates a reusuable connection based on config hash passed in</dd>
|
253
|
-
<dt>tables(schema: nil, catalog: nil)</dt>
|
254
|
-
<dd>returns a list of tables from the default connection or from the specified schema and catalog</dd>
|
255
|
-
<dt>metadata(table_name, schema: nil, catalog: nil)</dt>
|
256
|
-
<dd>provides metadata about a table</dd>
|
257
|
-
<dt>stats(table_name, date_column: nil)</dt>
|
258
|
-
<dd>provides table row count and date range</dd>
|
259
|
-
<dt>execute(sql, format: :array, retries: 0)</dt>
|
260
|
-
<dd>runs a query and returns in given format</dd>
|
261
|
-
<dt>execute_stream(sql, io, stats: nil)</dt>
|
262
|
-
<dd>runs a query and streams it as csv into the given io</dd>
|
263
|
-
</dl>
|
255
|
+
<p>execute_stream(sql, io, stats: nil)
|
256
|
+
: runs a query and streams it as csv into the given io</p>
|
264
257
|
|
265
258
|
<h2 id="error-handling">Error Handling</h2>
|
266
259
|
|
267
|
-
<
|
268
|
-
|
269
|
-
|
270
|
-
|
271
|
-
|
272
|
-
|
273
|
-
|
274
|
-
|
275
|
-
|
276
|
-
|
277
|
-
</code></p>
|
260
|
+
<pre class="code ruby"><code class="ruby"><span class='kw'>begin</span>
|
261
|
+
<span class='id identifier rubyid_results'>results</span> <span class='op'>=</span> <span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SELECT * FROM non_existent_table</span><span class='tstring_end'>"</span></span><span class='rparen'>)</span>
|
262
|
+
<span class='kw'>rescue</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="DWH/ExecutionError.html" title="DWH::ExecutionError (class)">ExecutionError</a></span></span> <span class='op'>=></span> <span class='id identifier rubyid_e'>e</span>
|
263
|
+
<span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Query failed: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_e'>e</span><span class='period'>.</span><span class='id identifier rubyid_message'>message</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
264
|
+
<span class='kw'>rescue</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="DWH/ConnectionError.html" title="DWH::ConnectionError (class)">ConnectionError</a></span></span> <span class='op'>=></span> <span class='id identifier rubyid_e'>e</span>
|
265
|
+
<span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Connection failed: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_e'>e</span><span class='period'>.</span><span class='id identifier rubyid_message'>message</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
266
|
+
<span class='kw'>rescue</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="DWH/ConfigError.html" title="DWH::ConfigError (class)">ConfigError</a></span></span> <span class='op'>=></span> <span class='id identifier rubyid_e'>e</span>
|
267
|
+
<span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Configuration error: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_e'>e</span><span class='period'>.</span><span class='id identifier rubyid_message'>message</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
268
|
+
<span class='kw'>end</span>
|
269
|
+
</code></pre>
|
278
270
|
</div></div>
|
279
271
|
|
280
272
|
<div id="footer">
|
281
|
-
Generated on
|
273
|
+
Generated on Mon Aug 25 10:59:27 2025 by
|
282
274
|
<a href="https://yardoc.org" title="Yay! A Ruby Documentation Tool" target="_parent">yard</a>
|
283
275
|
0.9.37 (ruby-3.4.4).
|
284
276
|
</div>
|