dwh 0.1.0 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +5 -4
- data/Rakefile +1 -1
- data/docs/DWH/Adapters/Adapter.html +33 -27
- data/docs/DWH/Adapters/Athena.html +25 -21
- data/docs/DWH/Adapters/Boolean.html +1 -1
- data/docs/DWH/Adapters/Druid.html +18 -12
- data/docs/DWH/Adapters/DuckDb.html +29 -27
- data/docs/DWH/Adapters/MySql.html +25 -19
- data/docs/DWH/Adapters/OpenAuthorizable/ClassMethods.html +3 -6
- data/docs/DWH/Adapters/OpenAuthorizable.html +5 -10
- data/docs/DWH/Adapters/Postgres.html +27 -23
- data/docs/DWH/Adapters/Snowflake.html +39 -24
- data/docs/DWH/Adapters/SqlServer.html +27 -25
- data/docs/DWH/Adapters/Trino.html +30 -30
- data/docs/DWH/Adapters.html +1 -1
- data/docs/DWH/AuthenticationError.html +1 -1
- data/docs/DWH/Behaviors.html +6 -11
- data/docs/DWH/Capabilities.html +10 -26
- data/docs/DWH/Column.html +7 -15
- data/docs/DWH/ConfigError.html +1 -1
- data/docs/DWH/ConnectionError.html +1 -1
- data/docs/DWH/DWHError.html +1 -1
- data/docs/DWH/ExecutionError.html +1 -1
- data/docs/DWH/Factory.html +1 -1
- data/docs/DWH/Functions/Arrays.html +8 -8
- data/docs/DWH/Functions/Dates.html +5 -7
- data/docs/DWH/Functions/ExtractDatePart.html +13 -25
- data/docs/DWH/Functions/Nulls.html +3 -3
- data/docs/DWH/Functions.html +6 -9
- data/docs/DWH/Logger.html +3 -5
- data/docs/DWH/OAuthError.html +1 -1
- data/docs/DWH/Settings.html +6 -9
- data/docs/DWH/StreamingStats.html +2 -3
- data/docs/DWH/Table.html +14 -26
- data/docs/DWH/TableStats.html +1 -1
- data/docs/DWH/TokenExpiredError.html +1 -1
- data/docs/DWH/UnsupportedCapability.html +1 -1
- data/docs/DWH.html +1 -1
- data/docs/_index.html +1 -1
- data/docs/file.README.html +43 -48
- data/docs/file.adapters.html +318 -343
- data/docs/file.creating-adapters.html +347 -357
- data/docs/file.getting-started.html +143 -151
- data/docs/file.usage.html +257 -278
- data/docs/index.html +43 -48
- data/docs/top-level-namespace.html +1 -1
- data/lib/dwh/adapters.rb +2 -2
- data/lib/dwh/version.rb +1 -1
- metadata +2 -2
@@ -68,426 +68,416 @@
|
|
68
68
|
<p>DWH adapters have a simple, focused architecture:</p>
|
69
69
|
|
70
70
|
<ul>
|
71
|
-
|
72
|
-
|
73
|
-
|
74
|
-
|
71
|
+
<li><strong>5 Core Methods</strong>: Every adapter must implement 5 essential methods</li>
|
72
|
+
<li><strong>YAML Settings</strong>: Database-specific behavior controlled by YAML configuration</li>
|
73
|
+
<li><strong>Configuration Validation</strong>: Automatic validation of connection parameters</li>
|
74
|
+
<li><strong>Function Translation</strong>: SQL functions automatically translated to database-specific syntax</li>
|
75
75
|
</ul>
|
76
76
|
|
77
77
|
<h2 id="minimal-adapter-example">Minimal Adapter Example</h2>
|
78
78
|
|
79
|
-
<p>Here
|
80
|
-
|
81
|
-
<
|
82
|
-
module
|
83
|
-
|
84
|
-
|
85
|
-
|
86
|
-
config
|
87
|
-
config
|
88
|
-
config
|
89
|
-
config
|
90
|
-
|
91
|
-
|
92
|
-
<
|
93
|
-
|
94
|
-
|
95
|
-
|
96
|
-
|
97
|
-
|
98
|
-
|
99
|
-
|
100
|
-
|
101
|
-
|
102
|
-
|
103
|
-
|
104
|
-
|
105
|
-
|
106
|
-
|
107
|
-
|
108
|
-
|
109
|
-
|
110
|
-
|
111
|
-
|
112
|
-
|
113
|
-
|
114
|
-
|
115
|
-
|
116
|
-
|
117
|
-
|
118
|
-
|
119
|
-
|
120
|
-
|
121
|
-
|
122
|
-
|
123
|
-
|
124
|
-
|
125
|
-
|
126
|
-
|
127
|
-
|
128
|
-
|
129
|
-
|
130
|
-
|
131
|
-
|
132
|
-
|
133
|
-
|
134
|
-
|
135
|
-
|
136
|
-
|
137
|
-
|
79
|
+
<p>Here's a minimal adapter implementation:</p>
|
80
|
+
|
81
|
+
<pre class="code ruby"><code class="ruby"><span class='kw'>module</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span>
|
82
|
+
<span class='kw'>module</span> <span class='const'>Adapters</span>
|
83
|
+
<span class='kw'>class</span> <span class='const'>MyCustomAdapter</span> <span class='op'><</span> <span class='const'>Adapter</span>
|
84
|
+
<span class='comment'># Define required configuration parameters
|
85
|
+
</span> <span class='id identifier rubyid_config'>config</span> <span class='symbol'>:host</span><span class='comma'>,</span> <span class='const'>String</span><span class='comma'>,</span> <span class='label'>required:</span> <span class='kw'>true</span><span class='comma'>,</span> <span class='label'>message:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>server host ip address or domain name</span><span class='tstring_end'>'</span></span>
|
86
|
+
<span class='id identifier rubyid_config'>config</span> <span class='symbol'>:port</span><span class='comma'>,</span> <span class='const'>Integer</span><span class='comma'>,</span> <span class='label'>required:</span> <span class='kw'>false</span><span class='comma'>,</span> <span class='label'>default:</span> <span class='int'>1234</span><span class='comma'>,</span> <span class='label'>message:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>port to connect to</span><span class='tstring_end'>'</span></span>
|
87
|
+
<span class='id identifier rubyid_config'>config</span> <span class='symbol'>:database</span><span class='comma'>,</span> <span class='const'>String</span><span class='comma'>,</span> <span class='label'>required:</span> <span class='kw'>true</span><span class='comma'>,</span> <span class='label'>message:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>name of database to connect to</span><span class='tstring_end'>'</span></span>
|
88
|
+
<span class='id identifier rubyid_config'>config</span> <span class='symbol'>:username</span><span class='comma'>,</span> <span class='const'>String</span><span class='comma'>,</span> <span class='label'>required:</span> <span class='kw'>true</span><span class='comma'>,</span> <span class='label'>message:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>connection username</span><span class='tstring_end'>'</span></span>
|
89
|
+
<span class='id identifier rubyid_config'>config</span> <span class='symbol'>:password</span><span class='comma'>,</span> <span class='const'>String</span><span class='comma'>,</span> <span class='label'>required:</span> <span class='kw'>false</span><span class='comma'>,</span> <span class='label'>default:</span> <span class='kw'>nil</span><span class='comma'>,</span> <span class='label'>message:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>connection password</span><span class='tstring_end'>'</span></span>
|
90
|
+
|
91
|
+
<span class='comment'># Implement required methods
|
92
|
+
</span> <span class='kw'>def</span> <span class='id identifier rubyid_connection'>connection</span>
|
93
|
+
<span class='comment'># Return your database connection object
|
94
|
+
</span> <span class='comment'># This is cached, so implement connection reuse here
|
95
|
+
</span> <span class='ivar'>@connection</span> <span class='op'>||=</span> <span class='id identifier rubyid_create_connection'>create_connection</span>
|
96
|
+
<span class='kw'>end</span>
|
97
|
+
|
98
|
+
<span class='kw'>def</span> <span class='id identifier rubyid_tables'>tables</span><span class='lparen'>(</span><span class='label'>catalog:</span> <span class='kw'>nil</span><span class='comma'>,</span> <span class='label'>schema:</span> <span class='kw'>nil</span><span class='rparen'>)</span>
|
99
|
+
<span class='comment'># Return array of DWH::Table objects
|
100
|
+
</span> <span class='comment'># Use catalog/schema for filtering if supported
|
101
|
+
</span> <span class='kw'>end</span>
|
102
|
+
|
103
|
+
<span class='kw'>def</span> <span class='id identifier rubyid_metadata'>metadata</span><span class='lparen'>(</span><span class='id identifier rubyid_table_name'>table_name</span><span class='comma'>,</span> <span class='label'>catalog:</span> <span class='kw'>nil</span><span class='comma'>,</span> <span class='label'>schema:</span> <span class='kw'>nil</span><span class='rparen'>)</span>
|
104
|
+
<span class='comment'># Return single DWH::Table object with column information
|
105
|
+
</span> <span class='kw'>end</span>
|
106
|
+
|
107
|
+
<span class='kw'>def</span> <span class='id identifier rubyid_stats'>stats</span><span class='lparen'>(</span><span class='id identifier rubyid_table_name'>table_name</span><span class='comma'>,</span> <span class='label'>date_column:</span> <span class='kw'>nil</span><span class='comma'>,</span> <span class='label'>catalog:</span> <span class='kw'>nil</span><span class='comma'>,</span> <span class='label'>schema:</span> <span class='kw'>nil</span><span class='rparen'>)</span>
|
108
|
+
<span class='comment'># Return DWH::TableStats object with row counts and date ranges
|
109
|
+
</span> <span class='kw'>end</span>
|
110
|
+
|
111
|
+
<span class='kw'>def</span> <span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='id identifier rubyid_sql'>sql</span><span class='comma'>,</span> <span class='label'>format:</span> <span class='symbol'>:array</span><span class='comma'>,</span> <span class='label'>retries:</span> <span class='int'>0</span><span class='rparen'>)</span>
|
112
|
+
<span class='comment'># Execute SQL and return results in specified format
|
113
|
+
</span> <span class='comment'># Formats: :array, :object, :csv, :native
|
114
|
+
</span> <span class='kw'>end</span>
|
115
|
+
|
116
|
+
<span class='kw'>def</span> <span class='id identifier rubyid_execute_stream'>execute_stream</span><span class='lparen'>(</span><span class='id identifier rubyid_sql'>sql</span><span class='comma'>,</span> <span class='id identifier rubyid_io'>io</span><span class='comma'>,</span> <span class='label'>stats:</span> <span class='kw'>nil</span><span class='rparen'>)</span>
|
117
|
+
<span class='comment'># Execute SQL and stream results directly to IO object
|
118
|
+
</span> <span class='kw'>end</span>
|
119
|
+
|
120
|
+
<span class='kw'>def</span> <span class='id identifier rubyid_stream'>stream</span><span class='lparen'>(</span><span class='id identifier rubyid_sql'>sql</span><span class='comma'>,</span> <span class='op'>&</span><span class='id identifier rubyid_block'>block</span><span class='rparen'>)</span>
|
121
|
+
<span class='comment'># Execute SQL and yield chunks to block
|
122
|
+
</span> <span class='kw'>end</span>
|
123
|
+
|
124
|
+
<span class='id identifier rubyid_private'>private</span>
|
125
|
+
|
126
|
+
<span class='kw'>def</span> <span class='id identifier rubyid_create_connection'>create_connection</span>
|
127
|
+
<span class='comment'># Your database-specific connection logic
|
128
|
+
</span> <span class='const'>MyDatabaseClient</span><span class='period'>.</span><span class='id identifier rubyid_connect'>connect</span><span class='lparen'>(</span>
|
129
|
+
<span class='label'>host:</span> <span class='id identifier rubyid_config'>config</span><span class='lbracket'>[</span><span class='symbol'>:host</span><span class='rbracket'>]</span><span class='comma'>,</span>
|
130
|
+
<span class='label'>port:</span> <span class='id identifier rubyid_config'>config</span><span class='lbracket'>[</span><span class='symbol'>:port</span><span class='rbracket'>]</span><span class='comma'>,</span>
|
131
|
+
<span class='label'>database:</span> <span class='id identifier rubyid_config'>config</span><span class='lbracket'>[</span><span class='symbol'>:database</span><span class='rbracket'>]</span><span class='comma'>,</span>
|
132
|
+
<span class='label'>username:</span> <span class='id identifier rubyid_config'>config</span><span class='lbracket'>[</span><span class='symbol'>:username</span><span class='rbracket'>]</span><span class='comma'>,</span>
|
133
|
+
<span class='label'>password:</span> <span class='id identifier rubyid_config'>config</span><span class='lbracket'>[</span><span class='symbol'>:password</span><span class='rbracket'>]</span>
|
134
|
+
<span class='rparen'>)</span>
|
135
|
+
<span class='kw'>end</span>
|
136
|
+
<span class='kw'>end</span>
|
137
|
+
<span class='kw'>end</span>
|
138
|
+
<span class='kw'>end</span>
|
139
|
+
|
140
|
+
<span class='comment'># Register your adapter
|
141
|
+
</span><span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='period'>.</span><span class='id identifier rubyid_register'><span class='object_link'><a href="DWH/Factory.html#register-instance_method" title="DWH::Factory#register (method)">register</a></span></span><span class='lparen'>(</span><span class='symbol'>:mycustom</span><span class='comma'>,</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="DWH/Adapters.html" title="DWH::Adapters (module)">Adapters</a></span></span><span class='op'>::</span><span class='const'>MyCustomAdapter</span><span class='rparen'>)</span>
|
138
142
|
</code></pre>
|
139
143
|
|
140
|
-
<h1 id="register-your-adapter">Register your adapter</h1>
|
141
|
-
<p>DWH.register(:mycustom, DWH::Adapters::MyCustomAdapter)
|
142
|
-
```</p>
|
143
|
-
|
144
144
|
<h2 id="step-by-step-implementation">Step-by-Step Implementation</h2>
|
145
145
|
|
146
|
-
<h3 id="define-configuration-parameters">1. Define Configuration Parameters</h3>
|
146
|
+
<h3 id="1-define-configuration-parameters">1. Define Configuration Parameters</h3>
|
147
147
|
|
148
148
|
<p>Use the <code>config</code> class method to define connection parameters:</p>
|
149
149
|
|
150
|
-
<
|
151
|
-
class
|
152
|
-
|
153
|
-
config
|
154
|
-
|
155
|
-
|
156
|
-
<
|
157
|
-
config
|
158
|
-
|
159
|
-
|
160
|
-
<
|
161
|
-
|
162
|
-
|
163
|
-
<
|
164
|
-
|
165
|
-
|
166
|
-
|
167
|
-
|
168
|
-
|
169
|
-
|
170
|
-
|
171
|
-
<
|
172
|
-
|
173
|
-
|
174
|
-
|
175
|
-
|
176
|
-
|
177
|
-
|
178
|
-
<
|
179
|
-
|
180
|
-
<
|
181
|
-
|
182
|
-
|
183
|
-
|
184
|
-
|
185
|
-
|
186
|
-
|
187
|
-
|
188
|
-
|
189
|
-
|
190
|
-
|
191
|
-
<
|
192
|
-
|
193
|
-
|
194
|
-
|
195
|
-
|
196
|
-
|
197
|
-
<
|
198
|
-
|
199
|
-
|
200
|
-
|
201
|
-
false
|
202
|
-
end
|
203
|
-
```</p>
|
204
|
-
|
205
|
-
<h3 id="implement-table-discovery">3. Implement Table Discovery</h3>
|
206
|
-
|
207
|
-
<p>```ruby
|
208
|
-
def tables(catalog: nil, schema: nil)
|
209
|
-
query = build_tables_query(catalog: catalog, schema: schema)
|
210
|
-
results = execute(query, format: :array)</p>
|
211
|
-
|
212
|
-
<p>results.map do |row|
|
213
|
-
DWH::Table.new(
|
214
|
-
physical_name: row[0],
|
215
|
-
schema: row[1] || ‘default’,
|
216
|
-
catalog: row[2],
|
217
|
-
table_type: row[3] || ‘TABLE’
|
218
|
-
)
|
219
|
-
end
|
220
|
-
end</p>
|
221
|
-
|
222
|
-
<p>private</p>
|
223
|
-
|
224
|
-
<p>def build_tables_query(catalog: nil, schema: nil)
|
225
|
-
query = “SHOW TABLES”</p>
|
226
|
-
|
227
|
-
<p>conditions = []
|
228
|
-
conditions « “FROM #catalog” if catalog
|
229
|
-
conditions « “LIKE ‘#schema.%’” if schema</p>
|
230
|
-
|
231
|
-
<p>query += “ #‘)” unless conditions.empty?
|
232
|
-
query
|
233
|
-
end
|
234
|
-
```</p>
|
235
|
-
|
236
|
-
<h3 id="implement-metadata-extraction">4. Implement Metadata Extraction</h3>
|
237
|
-
|
238
|
-
<p>```ruby
|
239
|
-
def metadata(table_name, catalog: nil, schema: nil)
|
240
|
-
# Parse table name if it includes schema/catalog
|
241
|
-
parsed = parse_table_name(table_name, catalog: catalog, schema: schema)</p>
|
242
|
-
|
243
|
-
<p>query = build_describe_query(parsed[:table], parsed[:schema], parsed[:catalog])
|
244
|
-
results = execute(query, format: :array)</p>
|
245
|
-
|
246
|
-
<p>columns = results.map do |row|
|
247
|
-
DWH::Column.new(
|
248
|
-
name: row[0],
|
249
|
-
data_type: row[1],
|
250
|
-
normalized_data_type: normalize_data_type(row[1]),
|
251
|
-
nullable: row[2] != ‘NO’,
|
252
|
-
default_value: row[3],
|
253
|
-
character_maximum_length: row[4],
|
254
|
-
numeric_precision: row[5],
|
255
|
-
numeric_scale: row[6]
|
256
|
-
)
|
257
|
-
end</p>
|
258
|
-
|
259
|
-
<p>DWH::Table.new(
|
260
|
-
physical_name: parsed[:table],
|
261
|
-
schema: parsed[:schema],
|
262
|
-
catalog: parsed[:catalog],
|
263
|
-
columns: columns
|
264
|
-
)
|
265
|
-
end
|
266
|
-
```</p>
|
267
|
-
|
268
|
-
<h3 id="implement-statistics-collection">5. Implement Statistics Collection</h3>
|
269
|
-
|
270
|
-
<p>```ruby
|
271
|
-
def stats(table_name, date_column: nil, catalog: nil, schema: nil)
|
272
|
-
parsed = parse_table_name(table_name, catalog: catalog, schema: schema)
|
273
|
-
full_table_name = build_full_table_name(parsed)</p>
|
274
|
-
|
275
|
-
<p># Get row count
|
276
|
-
count_query = “SELECT COUNT(*) FROM #full_table_name”
|
277
|
-
row_count = execute(count_query, format: :array).first.first</p>
|
278
|
-
|
279
|
-
<p># Get date range if date column provided
|
280
|
-
date_start = date_end = nil
|
281
|
-
if date_column
|
282
|
-
date_query = “SELECT MIN(#date_column), MAX(#date_column) FROM #full_table_name”
|
283
|
-
date_result = execute(date_query, format: :array).first
|
284
|
-
date_start, date_end = date_result
|
285
|
-
end</p>
|
286
|
-
|
287
|
-
<p>DWH::TableStats.new(
|
288
|
-
row_count: row_count,
|
289
|
-
date_start: date_start,
|
290
|
-
date_end: date_end
|
291
|
-
)
|
292
|
-
end
|
293
|
-
```</p>
|
294
|
-
|
295
|
-
<h3 id="implement-query-execution">6. Implement Query Execution</h3>
|
296
|
-
|
297
|
-
<p>```ruby
|
298
|
-
def execute(sql, format: :array, retries: 0)
|
299
|
-
response = connection.post(‘/query’, { sql: sql }.to_json)</p>
|
300
|
-
|
301
|
-
<p>raise DWH::ExecutionError, “Query failed: #responseresponse.body” unless response.success?</p>
|
302
|
-
|
303
|
-
<p>raw_data = JSON.parse(response.body)
|
304
|
-
format_results(raw_data, format)
|
305
|
-
rescue => e
|
306
|
-
if retries > 0
|
307
|
-
sleep(1)
|
308
|
-
execute(sql, format: format, retries: retries - 1)
|
309
|
-
else
|
310
|
-
raise DWH::ExecutionError, “Query execution failed: #ee.message”
|
311
|
-
end
|
312
|
-
end</p>
|
313
|
-
|
314
|
-
<p>def execute_stream(sql, io, stats: nil)
|
315
|
-
# For HTTP APIs, you might need to paginate or use streaming endpoints
|
316
|
-
offset = 0
|
317
|
-
limit = 10_000</p>
|
318
|
-
|
319
|
-
<p>loop do
|
320
|
-
paginated_sql = “#sql LIMIT #limit OFFSET #offset”
|
321
|
-
results = execute(paginated_sql, format: :array)</p>
|
322
|
-
|
323
|
-
<pre class="code ruby"><code class="ruby">break if results.empty?
|
324
|
-
|
325
|
-
results.each do |row|
|
326
|
-
csv_row = CSV.generate_line(row)
|
327
|
-
io.write(csv_row)
|
328
|
-
stats&.add_row(row)
|
329
|
-
end
|
330
|
-
|
331
|
-
offset += limit end end
|
150
|
+
<pre class="code ruby"><code class="ruby"><span class='kw'>class</span> <span class='const'>MyCustomAdapter</span> <span class='op'><</span> <span class='const'>Adapter</span>
|
151
|
+
<span class='comment'># Required parameters
|
152
|
+
</span> <span class='id identifier rubyid_config'>config</span> <span class='symbol'>:host</span><span class='comma'>,</span> <span class='const'>String</span><span class='comma'>,</span> <span class='label'>required:</span> <span class='kw'>true</span><span class='comma'>,</span> <span class='label'>message:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>server host ip address or domain name</span><span class='tstring_end'>'</span></span>
|
153
|
+
<span class='id identifier rubyid_config'>config</span> <span class='symbol'>:database</span><span class='comma'>,</span> <span class='const'>String</span><span class='comma'>,</span> <span class='label'>required:</span> <span class='kw'>true</span><span class='comma'>,</span> <span class='label'>message:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>name of database to connect to</span><span class='tstring_end'>'</span></span>
|
154
|
+
|
155
|
+
<span class='comment'># Optional parameters with defaults
|
156
|
+
</span> <span class='id identifier rubyid_config'>config</span> <span class='symbol'>:port</span><span class='comma'>,</span> <span class='const'>Integer</span><span class='comma'>,</span> <span class='label'>required:</span> <span class='kw'>false</span><span class='comma'>,</span> <span class='label'>default:</span> <span class='int'>5432</span><span class='comma'>,</span> <span class='label'>message:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>port to connect to</span><span class='tstring_end'>'</span></span>
|
157
|
+
<span class='id identifier rubyid_config'>config</span> <span class='symbol'>:timeout</span><span class='comma'>,</span> <span class='const'>Integer</span><span class='comma'>,</span> <span class='label'>required:</span> <span class='kw'>false</span><span class='comma'>,</span> <span class='label'>default:</span> <span class='int'>30</span><span class='comma'>,</span> <span class='label'>message:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>connection timeout</span><span class='tstring_end'>'</span></span>
|
158
|
+
|
159
|
+
<span class='comment'># Boolean parameters
|
160
|
+
</span> <span class='id identifier rubyid_config'>config</span> <span class='symbol'>:ssl</span><span class='comma'>,</span> <span class='const'>Boolean</span><span class='comma'>,</span> <span class='label'>required:</span> <span class='kw'>false</span><span class='comma'>,</span> <span class='label'>default:</span> <span class='kw'>false</span><span class='comma'>,</span> <span class='label'>message:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>use ssl connection</span><span class='tstring_end'>'</span></span>
|
161
|
+
|
162
|
+
<span class='comment'># Parameters with allowed values
|
163
|
+
</span> <span class='id identifier rubyid_config'>config</span> <span class='symbol'>:auth_type</span><span class='comma'>,</span> <span class='const'>String</span><span class='comma'>,</span> <span class='label'>required:</span> <span class='kw'>false</span><span class='comma'>,</span> <span class='label'>default:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>basic</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
164
|
+
<span class='label'>message:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>authentication type</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='label'>allowed:</span> <span class='qwords_beg'>%w[</span><span class='tstring_content'>basic</span><span class='words_sep'> </span><span class='tstring_content'>oauth</span><span class='words_sep'> </span><span class='tstring_content'>token</span><span class='tstring_end'>]</span></span>
|
165
|
+
<span class='kw'>end</span>
|
166
|
+
</code></pre>
|
167
|
+
|
168
|
+
<h3 id="2-implement-connection-management">2. Implement Connection Management</h3>
|
169
|
+
|
170
|
+
<pre class="code ruby"><code class="ruby"><span class='kw'>def</span> <span class='id identifier rubyid_connection'>connection</span>
|
171
|
+
<span class='kw'>return</span> <span class='ivar'>@connection</span> <span class='kw'>if</span> <span class='ivar'>@connection</span> <span class='op'>&&</span> <span class='id identifier rubyid_connection_valid?'>connection_valid?</span>
|
172
|
+
|
173
|
+
<span class='ivar'>@connection</span> <span class='op'>=</span> <span class='id identifier rubyid_create_connection'>create_connection</span>
|
174
|
+
<span class='kw'>end</span>
|
175
|
+
|
176
|
+
<span class='id identifier rubyid_private'>private</span>
|
177
|
+
|
178
|
+
<span class='kw'>def</span> <span class='id identifier rubyid_create_connection'>create_connection</span>
|
179
|
+
<span class='comment'># Example for HTTP-based database
|
180
|
+
</span> <span class='const'>Faraday</span><span class='period'>.</span><span class='id identifier rubyid_new'>new</span><span class='lparen'>(</span>
|
181
|
+
<span class='label'>url:</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_protocol'>protocol</span><span class='embexpr_end'>}</span><span class='tstring_content'>://</span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_config'>config</span><span class='lbracket'>[</span><span class='symbol'>:host</span><span class='rbracket'>]</span><span class='embexpr_end'>}</span><span class='tstring_content'>:</span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_config'>config</span><span class='lbracket'>[</span><span class='symbol'>:port</span><span class='rbracket'>]</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span><span class='comma'>,</span>
|
182
|
+
<span class='label'>headers:</span> <span class='id identifier rubyid_build_headers'>build_headers</span><span class='comma'>,</span>
|
183
|
+
<span class='label'>request:</span> <span class='lbrace'>{</span>
|
184
|
+
<span class='label'>timeout:</span> <span class='id identifier rubyid_config'>config</span><span class='lbracket'>[</span><span class='symbol'>:timeout</span><span class='rbracket'>]</span>
|
185
|
+
<span class='rbrace'>}</span>
|
186
|
+
<span class='rparen'>)</span>
|
187
|
+
<span class='kw'>end</span>
|
188
|
+
|
189
|
+
<span class='kw'>def</span> <span class='id identifier rubyid_build_headers'>build_headers</span>
|
190
|
+
<span class='id identifier rubyid_headers'>headers</span> <span class='op'>=</span> <span class='lbrace'>{</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>Content-Type</span><span class='tstring_end'>'</span></span> <span class='op'>=></span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>application/json</span><span class='tstring_end'>'</span></span> <span class='rbrace'>}</span>
|
191
|
+
<span class='id identifier rubyid_headers'>headers</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>Authorization</span><span class='tstring_end'>'</span></span><span class='rbracket'>]</span> <span class='op'>=</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Bearer </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_config'>config</span><span class='lbracket'>[</span><span class='symbol'>:token</span><span class='rbracket'>]</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span> <span class='kw'>if</span> <span class='id identifier rubyid_config'>config</span><span class='lbracket'>[</span><span class='symbol'>:token</span><span class='rbracket'>]</span>
|
192
|
+
<span class='id identifier rubyid_headers'>headers</span>
|
193
|
+
<span class='kw'>end</span>
|
194
|
+
|
195
|
+
<span class='kw'>def</span> <span class='id identifier rubyid_connection_valid?'>connection_valid?</span>
|
196
|
+
<span class='comment'># Implement connection health check
|
197
|
+
</span> <span class='ivar'>@connection</span><span class='op'>&.</span><span class='id identifier rubyid_get'>get</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>/health</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span><span class='op'>&.</span><span class='id identifier rubyid_success?'>success?</span>
|
198
|
+
<span class='kw'>rescue</span>
|
199
|
+
<span class='kw'>false</span>
|
200
|
+
<span class='kw'>end</span>
|
332
201
|
</code></pre>
|
333
202
|
|
334
|
-
<
|
335
|
-
|
336
|
-
|
337
|
-
|
203
|
+
<h3 id="3-implement-table-discovery">3. Implement Table Discovery</h3>
|
204
|
+
|
205
|
+
<pre class="code ruby"><code class="ruby"><span class='kw'>def</span> <span class='id identifier rubyid_tables'>tables</span><span class='lparen'>(</span><span class='label'>catalog:</span> <span class='kw'>nil</span><span class='comma'>,</span> <span class='label'>schema:</span> <span class='kw'>nil</span><span class='rparen'>)</span>
|
206
|
+
<span class='id identifier rubyid_query'>query</span> <span class='op'>=</span> <span class='id identifier rubyid_build_tables_query'>build_tables_query</span><span class='lparen'>(</span><span class='label'>catalog:</span> <span class='id identifier rubyid_catalog'>catalog</span><span class='comma'>,</span> <span class='label'>schema:</span> <span class='id identifier rubyid_schema'>schema</span><span class='rparen'>)</span>
|
207
|
+
<span class='id identifier rubyid_results'>results</span> <span class='op'>=</span> <span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='id identifier rubyid_query'>query</span><span class='comma'>,</span> <span class='label'>format:</span> <span class='symbol'>:array</span><span class='rparen'>)</span>
|
208
|
+
|
209
|
+
<span class='id identifier rubyid_results'>results</span><span class='period'>.</span><span class='id identifier rubyid_map'>map</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_row'>row</span><span class='op'>|</span>
|
210
|
+
<span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="DWH/Table.html" title="DWH::Table (class)">Table</a></span></span><span class='period'>.</span><span class='id identifier rubyid_new'><span class='object_link'><a href="DWH/Table.html#initialize-instance_method" title="DWH::Table#initialize (method)">new</a></span></span><span class='lparen'>(</span>
|
211
|
+
<span class='label'>physical_name:</span> <span class='id identifier rubyid_row'>row</span><span class='lbracket'>[</span><span class='int'>0</span><span class='rbracket'>]</span><span class='comma'>,</span>
|
212
|
+
<span class='label'>schema:</span> <span class='id identifier rubyid_row'>row</span><span class='lbracket'>[</span><span class='int'>1</span><span class='rbracket'>]</span> <span class='op'>||</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>default</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
213
|
+
<span class='label'>catalog:</span> <span class='id identifier rubyid_row'>row</span><span class='lbracket'>[</span><span class='int'>2</span><span class='rbracket'>]</span><span class='comma'>,</span>
|
214
|
+
<span class='label'>table_type:</span> <span class='id identifier rubyid_row'>row</span><span class='lbracket'>[</span><span class='int'>3</span><span class='rbracket'>]</span> <span class='op'>||</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>TABLE</span><span class='tstring_end'>'</span></span>
|
215
|
+
<span class='rparen'>)</span>
|
216
|
+
<span class='kw'>end</span>
|
217
|
+
<span class='kw'>end</span>
|
218
|
+
|
219
|
+
<span class='id identifier rubyid_private'>private</span>
|
220
|
+
|
221
|
+
<span class='kw'>def</span> <span class='id identifier rubyid_build_tables_query'>build_tables_query</span><span class='lparen'>(</span><span class='label'>catalog:</span> <span class='kw'>nil</span><span class='comma'>,</span> <span class='label'>schema:</span> <span class='kw'>nil</span><span class='rparen'>)</span>
|
222
|
+
<span class='id identifier rubyid_query'>query</span> <span class='op'>=</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SHOW TABLES</span><span class='tstring_end'>"</span></span>
|
338
223
|
|
339
|
-
<
|
340
|
-
|
341
|
-
|
224
|
+
<span class='id identifier rubyid_conditions'>conditions</span> <span class='op'>=</span> <span class='lbracket'>[</span><span class='rbracket'>]</span>
|
225
|
+
<span class='id identifier rubyid_conditions'>conditions</span> <span class='op'><<</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>FROM </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_catalog'>catalog</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span> <span class='kw'>if</span> <span class='id identifier rubyid_catalog'>catalog</span>
|
226
|
+
<span class='id identifier rubyid_conditions'>conditions</span> <span class='op'><<</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>LIKE '</span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_schema'>schema</span><span class='embexpr_end'>}</span><span class='tstring_content'>.%'</span><span class='tstring_end'>"</span></span> <span class='kw'>if</span> <span class='id identifier rubyid_schema'>schema</span>
|
342
227
|
|
343
|
-
<
|
228
|
+
<span class='id identifier rubyid_query'>query</span> <span class='op'>+=</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'> </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_conditions'>conditions</span><span class='period'>.</span><span class='id identifier rubyid_join'>join</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'> </span><span class='tstring_end'>'</span></span><span class='rparen'>)</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span> <span class='kw'>unless</span> <span class='id identifier rubyid_conditions'>conditions</span><span class='period'>.</span><span class='id identifier rubyid_empty?'>empty?</span>
|
229
|
+
<span class='id identifier rubyid_query'>query</span>
|
230
|
+
<span class='kw'>end</span>
|
231
|
+
</code></pre>
|
344
232
|
|
345
|
-
|
346
|
-
|
233
|
+
<h3 id="4-implement-metadata-extraction">4. Implement Metadata Extraction</h3>
|
234
|
+
|
235
|
+
<pre class="code ruby"><code class="ruby"><span class='kw'>def</span> <span class='id identifier rubyid_metadata'>metadata</span><span class='lparen'>(</span><span class='id identifier rubyid_table_name'>table_name</span><span class='comma'>,</span> <span class='label'>catalog:</span> <span class='kw'>nil</span><span class='comma'>,</span> <span class='label'>schema:</span> <span class='kw'>nil</span><span class='rparen'>)</span>
|
236
|
+
<span class='comment'># Parse table name if it includes schema/catalog
|
237
|
+
</span> <span class='id identifier rubyid_parsed'>parsed</span> <span class='op'>=</span> <span class='id identifier rubyid_parse_table_name'>parse_table_name</span><span class='lparen'>(</span><span class='id identifier rubyid_table_name'>table_name</span><span class='comma'>,</span> <span class='label'>catalog:</span> <span class='id identifier rubyid_catalog'>catalog</span><span class='comma'>,</span> <span class='label'>schema:</span> <span class='id identifier rubyid_schema'>schema</span><span class='rparen'>)</span>
|
238
|
+
|
239
|
+
<span class='id identifier rubyid_query'>query</span> <span class='op'>=</span> <span class='id identifier rubyid_build_describe_query'>build_describe_query</span><span class='lparen'>(</span><span class='id identifier rubyid_parsed'>parsed</span><span class='lbracket'>[</span><span class='symbol'>:table</span><span class='rbracket'>]</span><span class='comma'>,</span> <span class='id identifier rubyid_parsed'>parsed</span><span class='lbracket'>[</span><span class='symbol'>:schema</span><span class='rbracket'>]</span><span class='comma'>,</span> <span class='id identifier rubyid_parsed'>parsed</span><span class='lbracket'>[</span><span class='symbol'>:catalog</span><span class='rbracket'>]</span><span class='rparen'>)</span>
|
240
|
+
<span class='id identifier rubyid_results'>results</span> <span class='op'>=</span> <span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='id identifier rubyid_query'>query</span><span class='comma'>,</span> <span class='label'>format:</span> <span class='symbol'>:array</span><span class='rparen'>)</span>
|
241
|
+
|
242
|
+
<span class='id identifier rubyid_columns'>columns</span> <span class='op'>=</span> <span class='id identifier rubyid_results'>results</span><span class='period'>.</span><span class='id identifier rubyid_map'>map</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_row'>row</span><span class='op'>|</span>
|
243
|
+
<span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="DWH/Column.html" title="DWH::Column (class)">Column</a></span></span><span class='period'>.</span><span class='id identifier rubyid_new'><span class='object_link'><a href="DWH/Column.html#initialize-instance_method" title="DWH::Column#initialize (method)">new</a></span></span><span class='lparen'>(</span>
|
244
|
+
<span class='label'>name:</span> <span class='id identifier rubyid_row'>row</span><span class='lbracket'>[</span><span class='int'>0</span><span class='rbracket'>]</span><span class='comma'>,</span>
|
245
|
+
<span class='label'>data_type:</span> <span class='id identifier rubyid_row'>row</span><span class='lbracket'>[</span><span class='int'>1</span><span class='rbracket'>]</span><span class='comma'>,</span>
|
246
|
+
<span class='label'>normalized_data_type:</span> <span class='id identifier rubyid_normalize_data_type'>normalize_data_type</span><span class='lparen'>(</span><span class='id identifier rubyid_row'>row</span><span class='lbracket'>[</span><span class='int'>1</span><span class='rbracket'>]</span><span class='rparen'>)</span><span class='comma'>,</span>
|
247
|
+
<span class='label'>nullable:</span> <span class='id identifier rubyid_row'>row</span><span class='lbracket'>[</span><span class='int'>2</span><span class='rbracket'>]</span> <span class='op'>!=</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>NO</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
248
|
+
<span class='label'>default_value:</span> <span class='id identifier rubyid_row'>row</span><span class='lbracket'>[</span><span class='int'>3</span><span class='rbracket'>]</span><span class='comma'>,</span>
|
249
|
+
<span class='label'>character_maximum_length:</span> <span class='id identifier rubyid_row'>row</span><span class='lbracket'>[</span><span class='int'>4</span><span class='rbracket'>]</span><span class='comma'>,</span>
|
250
|
+
<span class='label'>numeric_precision:</span> <span class='id identifier rubyid_row'>row</span><span class='lbracket'>[</span><span class='int'>5</span><span class='rbracket'>]</span><span class='comma'>,</span>
|
251
|
+
<span class='label'>numeric_scale:</span> <span class='id identifier rubyid_row'>row</span><span class='lbracket'>[</span><span class='int'>6</span><span class='rbracket'>]</span>
|
252
|
+
<span class='rparen'>)</span>
|
253
|
+
<span class='kw'>end</span>
|
254
|
+
|
255
|
+
<span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="DWH/Table.html" title="DWH::Table (class)">Table</a></span></span><span class='period'>.</span><span class='id identifier rubyid_new'><span class='object_link'><a href="DWH/Table.html#initialize-instance_method" title="DWH::Table#initialize (method)">new</a></span></span><span class='lparen'>(</span>
|
256
|
+
<span class='label'>physical_name:</span> <span class='id identifier rubyid_parsed'>parsed</span><span class='lbracket'>[</span><span class='symbol'>:table</span><span class='rbracket'>]</span><span class='comma'>,</span>
|
257
|
+
<span class='label'>schema:</span> <span class='id identifier rubyid_parsed'>parsed</span><span class='lbracket'>[</span><span class='symbol'>:schema</span><span class='rbracket'>]</span><span class='comma'>,</span>
|
258
|
+
<span class='label'>catalog:</span> <span class='id identifier rubyid_parsed'>parsed</span><span class='lbracket'>[</span><span class='symbol'>:catalog</span><span class='rbracket'>]</span><span class='comma'>,</span>
|
259
|
+
<span class='label'>columns:</span> <span class='id identifier rubyid_columns'>columns</span>
|
260
|
+
<span class='rparen'>)</span>
|
261
|
+
<span class='kw'>end</span>
|
347
262
|
</code></pre>
|
348
263
|
|
349
|
-
<
|
350
|
-
|
351
|
-
<
|
352
|
-
|
353
|
-
|
354
|
-
|
355
|
-
|
356
|
-
|
357
|
-
|
358
|
-
|
359
|
-
|
360
|
-
|
361
|
-
|
362
|
-
|
363
|
-
|
364
|
-
|
365
|
-
|
366
|
-
|
367
|
-
|
368
|
-
|
264
|
+
<h3 id="5-implement-statistics-collection">5. Implement Statistics Collection</h3>
|
265
|
+
|
266
|
+
<pre class="code ruby"><code class="ruby"><span class='kw'>def</span> <span class='id identifier rubyid_stats'>stats</span><span class='lparen'>(</span><span class='id identifier rubyid_table_name'>table_name</span><span class='comma'>,</span> <span class='label'>date_column:</span> <span class='kw'>nil</span><span class='comma'>,</span> <span class='label'>catalog:</span> <span class='kw'>nil</span><span class='comma'>,</span> <span class='label'>schema:</span> <span class='kw'>nil</span><span class='rparen'>)</span>
|
267
|
+
<span class='id identifier rubyid_parsed'>parsed</span> <span class='op'>=</span> <span class='id identifier rubyid_parse_table_name'>parse_table_name</span><span class='lparen'>(</span><span class='id identifier rubyid_table_name'>table_name</span><span class='comma'>,</span> <span class='label'>catalog:</span> <span class='id identifier rubyid_catalog'>catalog</span><span class='comma'>,</span> <span class='label'>schema:</span> <span class='id identifier rubyid_schema'>schema</span><span class='rparen'>)</span>
|
268
|
+
<span class='id identifier rubyid_full_table_name'>full_table_name</span> <span class='op'>=</span> <span class='id identifier rubyid_build_full_table_name'>build_full_table_name</span><span class='lparen'>(</span><span class='id identifier rubyid_parsed'>parsed</span><span class='rparen'>)</span>
|
269
|
+
|
270
|
+
<span class='comment'># Get row count
|
271
|
+
</span> <span class='id identifier rubyid_count_query'>count_query</span> <span class='op'>=</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SELECT COUNT(*) FROM </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_full_table_name'>full_table_name</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
272
|
+
<span class='id identifier rubyid_row_count'>row_count</span> <span class='op'>=</span> <span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='id identifier rubyid_count_query'>count_query</span><span class='comma'>,</span> <span class='label'>format:</span> <span class='symbol'>:array</span><span class='rparen'>)</span><span class='period'>.</span><span class='id identifier rubyid_first'>first</span><span class='period'>.</span><span class='id identifier rubyid_first'>first</span>
|
273
|
+
|
274
|
+
<span class='comment'># Get date range if date column provided
|
275
|
+
</span> <span class='id identifier rubyid_date_start'>date_start</span> <span class='op'>=</span> <span class='id identifier rubyid_date_end'>date_end</span> <span class='op'>=</span> <span class='kw'>nil</span>
|
276
|
+
<span class='kw'>if</span> <span class='id identifier rubyid_date_column'>date_column</span>
|
277
|
+
<span class='id identifier rubyid_date_query'>date_query</span> <span class='op'>=</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SELECT MIN(</span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_date_column'>date_column</span><span class='embexpr_end'>}</span><span class='tstring_content'>), MAX(</span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_date_column'>date_column</span><span class='embexpr_end'>}</span><span class='tstring_content'>) FROM </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_full_table_name'>full_table_name</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
278
|
+
<span class='id identifier rubyid_date_result'>date_result</span> <span class='op'>=</span> <span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='id identifier rubyid_date_query'>date_query</span><span class='comma'>,</span> <span class='label'>format:</span> <span class='symbol'>:array</span><span class='rparen'>)</span><span class='period'>.</span><span class='id identifier rubyid_first'>first</span>
|
279
|
+
<span class='id identifier rubyid_date_start'>date_start</span><span class='comma'>,</span> <span class='id identifier rubyid_date_end'>date_end</span> <span class='op'>=</span> <span class='id identifier rubyid_date_result'>date_result</span>
|
280
|
+
<span class='kw'>end</span>
|
281
|
+
|
282
|
+
<span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="DWH/TableStats.html" title="DWH::TableStats (class)">TableStats</a></span></span><span class='period'>.</span><span class='id identifier rubyid_new'><span class='object_link'><a href="DWH/TableStats.html#initialize-instance_method" title="DWH::TableStats#initialize (method)">new</a></span></span><span class='lparen'>(</span>
|
283
|
+
<span class='label'>row_count:</span> <span class='id identifier rubyid_row_count'>row_count</span><span class='comma'>,</span>
|
284
|
+
<span class='label'>date_start:</span> <span class='id identifier rubyid_date_start'>date_start</span><span class='comma'>,</span>
|
285
|
+
<span class='label'>date_end:</span> <span class='id identifier rubyid_date_end'>date_end</span>
|
286
|
+
<span class='rparen'>)</span>
|
287
|
+
<span class='kw'>end</span>
|
288
|
+
</code></pre>
|
289
|
+
|
290
|
+
<h3 id="6-implement-query-execution">6. Implement Query Execution</h3>
|
291
|
+
|
292
|
+
<pre class="code ruby"><code class="ruby"><span class='kw'>def</span> <span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='id identifier rubyid_sql'>sql</span><span class='comma'>,</span> <span class='label'>format:</span> <span class='symbol'>:array</span><span class='comma'>,</span> <span class='label'>retries:</span> <span class='int'>0</span><span class='rparen'>)</span>
|
293
|
+
<span class='id identifier rubyid_response'>response</span> <span class='op'>=</span> <span class='id identifier rubyid_connection'>connection</span><span class='period'>.</span><span class='id identifier rubyid_post'>post</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>/query</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='lbrace'>{</span> <span class='label'>sql:</span> <span class='id identifier rubyid_sql'>sql</span> <span class='rbrace'>}</span><span class='period'>.</span><span class='id identifier rubyid_to_json'>to_json</span><span class='rparen'>)</span>
|
294
|
+
|
295
|
+
<span class='id identifier rubyid_raise'>raise</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="DWH/ExecutionError.html" title="DWH::ExecutionError (class)">ExecutionError</a></span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Query failed: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_response'>response</span><span class='period'>.</span><span class='id identifier rubyid_body'>body</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span> <span class='kw'>unless</span> <span class='id identifier rubyid_response'>response</span><span class='period'>.</span><span class='id identifier rubyid_success?'>success?</span>
|
296
|
+
|
297
|
+
<span class='id identifier rubyid_raw_data'>raw_data</span> <span class='op'>=</span> <span class='const'>JSON</span><span class='period'>.</span><span class='id identifier rubyid_parse'>parse</span><span class='lparen'>(</span><span class='id identifier rubyid_response'>response</span><span class='period'>.</span><span class='id identifier rubyid_body'>body</span><span class='rparen'>)</span>
|
298
|
+
<span class='id identifier rubyid_format_results'>format_results</span><span class='lparen'>(</span><span class='id identifier rubyid_raw_data'>raw_data</span><span class='comma'>,</span> <span class='id identifier rubyid_format'>format</span><span class='rparen'>)</span>
|
299
|
+
<span class='kw'>rescue</span> <span class='op'>=></span> <span class='id identifier rubyid_e'>e</span>
|
300
|
+
<span class='kw'>if</span> <span class='id identifier rubyid_retries'>retries</span> <span class='op'>></span> <span class='int'>0</span>
|
301
|
+
<span class='id identifier rubyid_sleep'>sleep</span><span class='lparen'>(</span><span class='int'>1</span><span class='rparen'>)</span>
|
302
|
+
<span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='id identifier rubyid_sql'>sql</span><span class='comma'>,</span> <span class='label'>format:</span> <span class='id identifier rubyid_format'>format</span><span class='comma'>,</span> <span class='label'>retries:</span> <span class='id identifier rubyid_retries'>retries</span> <span class='op'>-</span> <span class='int'>1</span><span class='rparen'>)</span>
|
303
|
+
<span class='kw'>else</span>
|
304
|
+
<span class='id identifier rubyid_raise'>raise</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="DWH/ExecutionError.html" title="DWH::ExecutionError (class)">ExecutionError</a></span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Query execution failed: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_e'>e</span><span class='period'>.</span><span class='id identifier rubyid_message'>message</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
305
|
+
<span class='kw'>end</span>
|
306
|
+
<span class='kw'>end</span>
|
307
|
+
|
308
|
+
<span class='kw'>def</span> <span class='id identifier rubyid_execute_stream'>execute_stream</span><span class='lparen'>(</span><span class='id identifier rubyid_sql'>sql</span><span class='comma'>,</span> <span class='id identifier rubyid_io'>io</span><span class='comma'>,</span> <span class='label'>stats:</span> <span class='kw'>nil</span><span class='rparen'>)</span>
|
309
|
+
<span class='comment'># For HTTP APIs, you might need to paginate or use streaming endpoints
|
310
|
+
</span> <span class='id identifier rubyid_offset'>offset</span> <span class='op'>=</span> <span class='int'>0</span>
|
311
|
+
<span class='id identifier rubyid_limit'>limit</span> <span class='op'>=</span> <span class='int'>10_000</span>
|
312
|
+
|
313
|
+
<span class='id identifier rubyid_loop'>loop</span> <span class='kw'>do</span>
|
314
|
+
<span class='id identifier rubyid_paginated_sql'>paginated_sql</span> <span class='op'>=</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_sql'>sql</span><span class='embexpr_end'>}</span><span class='tstring_content'> LIMIT </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_limit'>limit</span><span class='embexpr_end'>}</span><span class='tstring_content'> OFFSET </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_offset'>offset</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
315
|
+
<span class='id identifier rubyid_results'>results</span> <span class='op'>=</span> <span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='id identifier rubyid_paginated_sql'>paginated_sql</span><span class='comma'>,</span> <span class='label'>format:</span> <span class='symbol'>:array</span><span class='rparen'>)</span>
|
316
|
+
|
317
|
+
<span class='kw'>break</span> <span class='kw'>if</span> <span class='id identifier rubyid_results'>results</span><span class='period'>.</span><span class='id identifier rubyid_empty?'>empty?</span>
|
318
|
+
|
319
|
+
<span class='id identifier rubyid_results'>results</span><span class='period'>.</span><span class='id identifier rubyid_each'>each</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_row'>row</span><span class='op'>|</span>
|
320
|
+
<span class='id identifier rubyid_csv_row'>csv_row</span> <span class='op'>=</span> <span class='const'>CSV</span><span class='period'>.</span><span class='id identifier rubyid_generate_line'>generate_line</span><span class='lparen'>(</span><span class='id identifier rubyid_row'>row</span><span class='rparen'>)</span>
|
321
|
+
<span class='id identifier rubyid_io'>io</span><span class='period'>.</span><span class='id identifier rubyid_write'>write</span><span class='lparen'>(</span><span class='id identifier rubyid_csv_row'>csv_row</span><span class='rparen'>)</span>
|
322
|
+
<span class='id identifier rubyid_stats'>stats</span><span class='op'>&.</span><span class='id identifier rubyid_add_row'>add_row</span><span class='lparen'>(</span><span class='id identifier rubyid_row'>row</span><span class='rparen'>)</span>
|
323
|
+
<span class='kw'>end</span>
|
324
|
+
|
325
|
+
<span class='id identifier rubyid_offset'>offset</span> <span class='op'>+=</span> <span class='id identifier rubyid_limit'>limit</span>
|
326
|
+
<span class='kw'>end</span>
|
327
|
+
<span class='kw'>end</span>
|
328
|
+
|
329
|
+
<span class='kw'>def</span> <span class='id identifier rubyid_stream'>stream</span><span class='lparen'>(</span><span class='id identifier rubyid_sql'>sql</span><span class='comma'>,</span> <span class='op'>&</span><span class='id identifier rubyid_block'>block</span><span class='rparen'>)</span>
|
330
|
+
<span class='comment'># Similar to execute_stream but yields chunks to block
|
331
|
+
</span> <span class='id identifier rubyid_offset'>offset</span> <span class='op'>=</span> <span class='int'>0</span>
|
332
|
+
<span class='id identifier rubyid_limit'>limit</span> <span class='op'>=</span> <span class='int'>10_000</span>
|
333
|
+
|
334
|
+
<span class='id identifier rubyid_loop'>loop</span> <span class='kw'>do</span>
|
335
|
+
<span class='id identifier rubyid_paginated_sql'>paginated_sql</span> <span class='op'>=</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_sql'>sql</span><span class='embexpr_end'>}</span><span class='tstring_content'> LIMIT </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_limit'>limit</span><span class='embexpr_end'>}</span><span class='tstring_content'> OFFSET </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_offset'>offset</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
336
|
+
<span class='id identifier rubyid_chunk'>chunk</span> <span class='op'>=</span> <span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='id identifier rubyid_paginated_sql'>paginated_sql</span><span class='comma'>,</span> <span class='label'>format:</span> <span class='symbol'>:array</span><span class='rparen'>)</span>
|
337
|
+
|
338
|
+
<span class='kw'>break</span> <span class='kw'>if</span> <span class='id identifier rubyid_chunk'>chunk</span><span class='period'>.</span><span class='id identifier rubyid_empty?'>empty?</span>
|
339
|
+
|
340
|
+
<span class='kw'>yield</span> <span class='id identifier rubyid_chunk'>chunk</span>
|
341
|
+
<span class='id identifier rubyid_offset'>offset</span> <span class='op'>+=</span> <span class='id identifier rubyid_limit'>limit</span>
|
342
|
+
<span class='kw'>end</span>
|
343
|
+
<span class='kw'>end</span>
|
344
|
+
|
345
|
+
<span class='id identifier rubyid_private'>private</span>
|
346
|
+
|
347
|
+
<span class='kw'>def</span> <span class='id identifier rubyid_format_results'>format_results</span><span class='lparen'>(</span><span class='id identifier rubyid_raw_data'>raw_data</span><span class='comma'>,</span> <span class='id identifier rubyid_format'>format</span><span class='rparen'>)</span>
|
348
|
+
<span class='kw'>case</span> <span class='id identifier rubyid_format'>format</span>
|
349
|
+
<span class='kw'>when</span> <span class='symbol'>:array</span>
|
350
|
+
<span class='id identifier rubyid_raw_data'>raw_data</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>rows</span><span class='tstring_end'>'</span></span><span class='rbracket'>]</span>
|
351
|
+
<span class='kw'>when</span> <span class='symbol'>:object</span>
|
352
|
+
<span class='id identifier rubyid_columns'>columns</span> <span class='op'>=</span> <span class='id identifier rubyid_raw_data'>raw_data</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>columns</span><span class='tstring_end'>'</span></span><span class='rbracket'>]</span>
|
353
|
+
<span class='id identifier rubyid_raw_data'>raw_data</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>rows</span><span class='tstring_end'>'</span></span><span class='rbracket'>]</span><span class='period'>.</span><span class='id identifier rubyid_map'>map</span> <span class='lbrace'>{</span> <span class='op'>|</span><span class='id identifier rubyid_row'>row</span><span class='op'>|</span> <span class='id identifier rubyid_columns'>columns</span><span class='period'>.</span><span class='id identifier rubyid_zip'>zip</span><span class='lparen'>(</span><span class='id identifier rubyid_row'>row</span><span class='rparen'>)</span><span class='period'>.</span><span class='id identifier rubyid_to_h'>to_h</span> <span class='rbrace'>}</span>
|
354
|
+
<span class='kw'>when</span> <span class='symbol'>:csv</span>
|
355
|
+
<span class='const'>CSV</span><span class='period'>.</span><span class='id identifier rubyid_generate'>generate</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_csv'>csv</span><span class='op'>|</span>
|
356
|
+
<span class='id identifier rubyid_raw_data'>raw_data</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>rows</span><span class='tstring_end'>'</span></span><span class='rbracket'>]</span><span class='period'>.</span><span class='id identifier rubyid_each'>each</span> <span class='lbrace'>{</span> <span class='op'>|</span><span class='id identifier rubyid_row'>row</span><span class='op'>|</span> <span class='id identifier rubyid_csv'>csv</span> <span class='op'><<</span> <span class='id identifier rubyid_row'>row</span> <span class='rbrace'>}</span>
|
357
|
+
<span class='kw'>end</span>
|
358
|
+
<span class='kw'>when</span> <span class='symbol'>:native</span>
|
359
|
+
<span class='id identifier rubyid_raw_data'>raw_data</span>
|
360
|
+
<span class='kw'>else</span>
|
361
|
+
<span class='id identifier rubyid_raise'>raise</span> <span class='const'>ArgumentError</span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Unsupported format: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_format'>format</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
362
|
+
<span class='kw'>end</span>
|
363
|
+
<span class='kw'>end</span>
|
364
|
+
</code></pre>
|
369
365
|
|
370
366
|
<h2 id="creating-custom-settings">Creating Custom Settings</h2>
|
371
367
|
|
372
|
-
<h3 id="create-settings-file">1. Create Settings File</h3>
|
368
|
+
<h3 id="1-create-settings-file">1. Create Settings File</h3>
|
373
369
|
|
374
370
|
<p>Create by copying the <a href="https://github.com/stratasite/dwh/blob/main/lib/dwh/settings/base.yml">base settings file</a> to a relative directory like so:<code>settings/mycustom.yml</code></p>
|
375
371
|
|
376
|
-
<
|
377
|
-
# Override base settings for your database</p>
|
372
|
+
<pre class="code yaml"><code class="yaml"># Override base settings for your database
|
378
373
|
|
379
|
-
|
380
|
-
|
381
|
-
date_literal:
|
382
|
-
cast:
|
374
|
+
# Function mappings
|
375
|
+
truncate_date: "DATE_TRUNC('@unit', @exp)"
|
376
|
+
date_literal: "DATE('@val')"
|
377
|
+
cast: "CAST(@exp AS @type)"
|
383
378
|
|
384
|
-
|
385
|
-
|
386
|
-
upper_case:
|
387
|
-
lower_case:
|
379
|
+
# String functions
|
380
|
+
trim: "LTRIM(RTRIM(@exp))"
|
381
|
+
upper_case: "UPPER(@exp)"
|
382
|
+
lower_case: "LOWER(@exp)"
|
388
383
|
|
389
|
-
|
390
|
-
|
391
|
-
null_if:
|
384
|
+
# Null handling
|
385
|
+
if_null: "ISNULL(@exp, @when_null)"
|
386
|
+
null_if: "CASE WHEN @exp = @target THEN NULL ELSE @exp END"
|
392
387
|
|
393
|
-
|
394
|
-
|
388
|
+
# Capabilities
|
389
|
+
supports_window_functions: true
|
395
390
|
supports_array_functions: false
|
396
391
|
supports_common_table_expressions: true
|
397
|
-
supports_temp_tables: false
|
392
|
+
supports_temp_tables: false
|
398
393
|
|
399
|
-
|
400
|
-
|
401
|
-
final_pass_measure_join_type:
|
394
|
+
# Query behavior
|
395
|
+
temp_table_type: "subquery" # options: cte, subquery, temp
|
396
|
+
final_pass_measure_join_type: "inner" # inner, left, right, full
|
402
397
|
|
403
|
-
|
404
|
-
|
398
|
+
# Custom settings for your database
|
399
|
+
custom_query_prefix: "/* Generated by DWH */"
|
405
400
|
max_query_length: 1000000
|
406
|
-
|
401
|
+
</code></pre>
|
407
402
|
|
408
|
-
<h3 id="custom-settings-location">2. Custom Settings Location</h3>
|
403
|
+
<h3 id="2-custom-settings-location">2. Custom Settings Location</h3>
|
409
404
|
|
410
|
-
<
|
411
|
-
class
|
412
|
-
|
413
|
-
settings_file_path “/path/to/my_custom_settings.yml”</p>
|
405
|
+
<pre class="code ruby"><code class="ruby"><span class='kw'>class</span> <span class='const'>MyCustomAdapter</span> <span class='op'><</span> <span class='const'>Adapter</span>
|
406
|
+
<span class='comment'># Specify custom settings file location
|
407
|
+
</span> <span class='id identifier rubyid_settings_file_path'>settings_file_path</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>/path/to/my_custom_settings.yml</span><span class='tstring_end'>"</span></span>
|
414
408
|
|
415
|
-
<
|
416
|
-
end
|
417
|
-
|
409
|
+
<span class='comment'># ... rest of implementation
|
410
|
+
</span><span class='kw'>end</span>
|
411
|
+
</code></pre>
|
418
412
|
|
419
413
|
<h2 id="advanced-features">Advanced Features</h2>
|
420
414
|
|
421
415
|
<h3 id="error-handling">Error Handling</h3>
|
422
416
|
|
423
|
-
<
|
424
|
-
|
425
|
-
|
426
|
-
|
427
|
-
|
428
|
-
|
429
|
-
|
430
|
-
|
431
|
-
|
432
|
-
|
433
|
-
</code></p>
|
417
|
+
<pre class="code ruby"><code class="ruby"><span class='kw'>def</span> <span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='id identifier rubyid_sql'>sql</span><span class='comma'>,</span> <span class='label'>format:</span> <span class='symbol'>:array</span><span class='comma'>,</span> <span class='label'>retries:</span> <span class='int'>0</span><span class='rparen'>)</span>
|
418
|
+
<span class='comment'># Your execution logic
|
419
|
+
</span><span class='kw'>rescue</span> <span class='const'>MyDatabaseClient</span><span class='op'>::</span><span class='const'>ConnectionError</span> <span class='op'>=></span> <span class='id identifier rubyid_e'>e</span>
|
420
|
+
<span class='id identifier rubyid_raise'>raise</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="DWH/ConnectionError.html" title="DWH::ConnectionError (class)">ConnectionError</a></span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Database connection failed: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_e'>e</span><span class='period'>.</span><span class='id identifier rubyid_message'>message</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
421
|
+
<span class='kw'>rescue</span> <span class='const'>MyDatabaseClient</span><span class='op'>::</span><span class='const'>QueryError</span> <span class='op'>=></span> <span class='id identifier rubyid_e'>e</span>
|
422
|
+
<span class='id identifier rubyid_raise'>raise</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="DWH/ExecutionError.html" title="DWH::ExecutionError (class)">ExecutionError</a></span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Query execution failed: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_e'>e</span><span class='period'>.</span><span class='id identifier rubyid_message'>message</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
423
|
+
<span class='kw'>rescue</span> <span class='op'>=></span> <span class='id identifier rubyid_e'>e</span>
|
424
|
+
<span class='id identifier rubyid_raise'>raise</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='op'>::</span><span class='const'>AdapterError</span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Unexpected error: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_e'>e</span><span class='period'>.</span><span class='id identifier rubyid_message'>message</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
425
|
+
<span class='kw'>end</span>
|
426
|
+
</code></pre>
|
434
427
|
|
435
428
|
<h3 id="custom-function-translation">Custom Function Translation</h3>
|
436
429
|
|
437
|
-
<
|
438
|
-
|
439
|
-
|
440
|
-
template = settings[:custom_function] || “CUSTOM_FUNC(@exp, @p1, @p2)”</p>
|
430
|
+
<pre class="code ruby"><code class="ruby"><span class='kw'>def</span> <span class='id identifier rubyid_custom_function'>custom_function</span><span class='lparen'>(</span><span class='id identifier rubyid_expression'>expression</span><span class='comma'>,</span> <span class='id identifier rubyid_param1'>param1</span><span class='comma'>,</span> <span class='id identifier rubyid_param2'>param2</span><span class='rparen'>)</span>
|
431
|
+
<span class='comment'># Access settings for function templates
|
432
|
+
</span> <span class='id identifier rubyid_template'>template</span> <span class='op'>=</span> <span class='id identifier rubyid_settings'>settings</span><span class='lbracket'>[</span><span class='symbol'>:custom_function</span><span class='rbracket'>]</span> <span class='op'>||</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>CUSTOM_FUNC(@exp, @p1, @p2)</span><span class='tstring_end'>"</span></span>
|
441
433
|
|
442
|
-
<
|
443
|
-
|
444
|
-
|
445
|
-
end
|
446
|
-
|
434
|
+
<span class='id identifier rubyid_template'>template</span><span class='period'>.</span><span class='id identifier rubyid_gsub'>gsub</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>@exp</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='id identifier rubyid_expression'>expression</span><span class='rparen'>)</span>
|
435
|
+
<span class='period'>.</span><span class='id identifier rubyid_gsub'>gsub</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>@p1</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='id identifier rubyid_param1'>param1</span><span class='period'>.</span><span class='id identifier rubyid_to_s'>to_s</span><span class='rparen'>)</span>
|
436
|
+
<span class='period'>.</span><span class='id identifier rubyid_gsub'>gsub</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>@p2</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='id identifier rubyid_param2'>param2</span><span class='period'>.</span><span class='id identifier rubyid_to_s'>to_s</span><span class='rparen'>)</span>
|
437
|
+
<span class='kw'>end</span>
|
438
|
+
</code></pre>
|
447
439
|
|
448
440
|
<h2 id="registration-and-usage">Registration and Usage</h2>
|
449
441
|
|
450
|
-
<h3 id="register-your-adapter
|
442
|
+
<h3 id="register-your-adapter">Register Your Adapter</h3>
|
451
443
|
|
452
|
-
<
|
453
|
-
|
454
|
-
require
|
455
|
-
require ‘my_custom_adapter’</p>
|
444
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># In your gem or application initialization
|
445
|
+
</span><span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>dwh</span><span class='tstring_end'>'</span></span>
|
446
|
+
<span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>my_custom_adapter</span><span class='tstring_end'>'</span></span>
|
456
447
|
|
457
|
-
<
|
458
|
-
|
448
|
+
<span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='period'>.</span><span class='id identifier rubyid_register'><span class='object_link'><a href="DWH/Factory.html#register-instance_method" title="DWH::Factory#register (method)">register</a></span></span><span class='lparen'>(</span><span class='symbol'>:mycustom</span><span class='comma'>,</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="DWH/Adapters.html" title="DWH::Adapters (module)">Adapters</a></span></span><span class='op'>::</span><span class='const'>MyCustomAdapter</span><span class='rparen'>)</span>
|
449
|
+
</code></pre>
|
459
450
|
|
460
451
|
<h3 id="use-your-adapter">Use Your Adapter</h3>
|
461
452
|
|
462
|
-
<
|
463
|
-
|
464
|
-
|
465
|
-
|
466
|
-
|
467
|
-
|
468
|
-
|
469
|
-
|
470
|
-
|
471
|
-
|
472
|
-
<
|
473
|
-
<
|
474
|
-
|
475
|
-
|
476
|
-
```</p>
|
453
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Create adapter instance
|
454
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span> <span class='op'>=</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='period'>.</span><span class='id identifier rubyid_create'><span class='object_link'><a href="DWH/Factory.html#create-instance_method" title="DWH::Factory#create (method)">create</a></span></span><span class='lparen'>(</span><span class='symbol'>:mycustom</span><span class='comma'>,</span> <span class='lbrace'>{</span>
|
455
|
+
<span class='label'>host:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>database.example.com</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
456
|
+
<span class='label'>port:</span> <span class='int'>1234</span><span class='comma'>,</span>
|
457
|
+
<span class='label'>database:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>analytics</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
458
|
+
<span class='label'>username:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>analyst</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
459
|
+
<span class='label'>password:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>secret</span><span class='tstring_end'>'</span></span>
|
460
|
+
<span class='rbrace'>}</span><span class='rparen'>)</span>
|
461
|
+
|
462
|
+
<span class='comment'># Use standard DWH interface
|
463
|
+
</span><span class='id identifier rubyid_tables'>tables</span> <span class='op'>=</span> <span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_tables'>tables</span>
|
464
|
+
<span class='id identifier rubyid_metadata'>metadata</span> <span class='op'>=</span> <span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_metadata'>metadata</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>users</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span>
|
465
|
+
<span class='id identifier rubyid_results'>results</span> <span class='op'>=</span> <span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SELECT COUNT(*) FROM users</span><span class='tstring_end'>"</span></span><span class='rparen'>)</span>
|
466
|
+
</code></pre>
|
477
467
|
|
478
468
|
<h2 id="examples-to-study">Examples to Study</h2>
|
479
469
|
|
480
470
|
<p>Look at existing adapters for implementation patterns:</p>
|
481
471
|
|
482
472
|
<ul>
|
483
|
-
|
484
|
-
|
485
|
-
|
473
|
+
<li><strong>PostgreSQL</strong> (<code>lib/dwh/adapters/postgres.rb</code>) - RDBMS with full SQL support</li>
|
474
|
+
<li><strong>Druid</strong> (<code>lib/dwh/adapters/druid.rb</code>) - HTTP API-based adapter</li>
|
475
|
+
<li><strong>DuckDB</strong> (<code>lib/dwh/adapters/duck_db.rb</code>) - Embedded database adapter</li>
|
486
476
|
</ul>
|
487
477
|
</div></div>
|
488
478
|
|
489
479
|
<div id="footer">
|
490
|
-
Generated on
|
480
|
+
Generated on Mon Aug 25 10:59:27 2025 by
|
491
481
|
<a href="https://yardoc.org" title="Yay! A Ruby Documentation Tool" target="_parent">yard</a>
|
492
482
|
0.9.37 (ruby-3.4.4).
|
493
483
|
</div>
|