dwh 0.1.0 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +5 -4
- data/Rakefile +1 -1
- data/docs/DWH/Adapters/Adapter.html +33 -27
- data/docs/DWH/Adapters/Athena.html +25 -21
- data/docs/DWH/Adapters/Boolean.html +1 -1
- data/docs/DWH/Adapters/Druid.html +18 -12
- data/docs/DWH/Adapters/DuckDb.html +29 -27
- data/docs/DWH/Adapters/MySql.html +25 -19
- data/docs/DWH/Adapters/OpenAuthorizable/ClassMethods.html +3 -6
- data/docs/DWH/Adapters/OpenAuthorizable.html +5 -10
- data/docs/DWH/Adapters/Postgres.html +27 -23
- data/docs/DWH/Adapters/Snowflake.html +39 -24
- data/docs/DWH/Adapters/SqlServer.html +27 -25
- data/docs/DWH/Adapters/Trino.html +30 -30
- data/docs/DWH/Adapters.html +1 -1
- data/docs/DWH/AuthenticationError.html +1 -1
- data/docs/DWH/Behaviors.html +6 -11
- data/docs/DWH/Capabilities.html +10 -26
- data/docs/DWH/Column.html +7 -15
- data/docs/DWH/ConfigError.html +1 -1
- data/docs/DWH/ConnectionError.html +1 -1
- data/docs/DWH/DWHError.html +1 -1
- data/docs/DWH/ExecutionError.html +1 -1
- data/docs/DWH/Factory.html +1 -1
- data/docs/DWH/Functions/Arrays.html +8 -8
- data/docs/DWH/Functions/Dates.html +5 -7
- data/docs/DWH/Functions/ExtractDatePart.html +13 -25
- data/docs/DWH/Functions/Nulls.html +3 -3
- data/docs/DWH/Functions.html +6 -9
- data/docs/DWH/Logger.html +3 -5
- data/docs/DWH/OAuthError.html +1 -1
- data/docs/DWH/Settings.html +6 -9
- data/docs/DWH/StreamingStats.html +2 -3
- data/docs/DWH/Table.html +14 -26
- data/docs/DWH/TableStats.html +1 -1
- data/docs/DWH/TokenExpiredError.html +1 -1
- data/docs/DWH/UnsupportedCapability.html +1 -1
- data/docs/DWH.html +1 -1
- data/docs/_index.html +1 -1
- data/docs/file.README.html +43 -48
- data/docs/file.adapters.html +318 -343
- data/docs/file.creating-adapters.html +347 -357
- data/docs/file.getting-started.html +143 -151
- data/docs/file.usage.html +257 -278
- data/docs/index.html +43 -48
- data/docs/top-level-namespace.html +1 -1
- data/lib/dwh/adapters.rb +2 -2
- data/lib/dwh/version.rb +1 -1
- metadata +2 -2
data/docs/file.usage.html
CHANGED
@@ -69,374 +69,353 @@
|
|
69
69
|
|
70
70
|
<h4 id="date-truncation">Date Truncation</h4>
|
71
71
|
|
72
|
-
<
|
73
|
-
|
74
|
-
|
75
|
-
|
76
|
-
|
77
|
-
|
78
|
-
|
79
|
-
<
|
80
|
-
|
81
|
-
```</p>
|
72
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Truncate dates to different periods
|
73
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_truncate_date'>truncate_date</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>week</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>created_at</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span>
|
74
|
+
<span class='comment'># PostgreSQL: DATE_TRUNC('week', created_at)
|
75
|
+
</span><span class='comment'># SQL Server: DATETRUNC(week, created_at)
|
76
|
+
</span><span class='comment'># MySQL: DATE_FORMAT(created_at, '%Y-%m-%d') - complex logic
|
77
|
+
</span>
|
78
|
+
<span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_truncate_date'>truncate_date</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>month</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>order_date</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span>
|
79
|
+
<span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_truncate_date'>truncate_date</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>year</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>signup_date</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span>
|
80
|
+
</code></pre>
|
82
81
|
|
83
82
|
<h4 id="date-literals">Date Literals</h4>
|
84
83
|
|
85
|
-
<
|
86
|
-
|
87
|
-
|
88
|
-
|
89
|
-
|
90
|
-
|
91
|
-
|
92
|
-
<
|
93
|
-
|
94
|
-
|
95
|
-
```</p>
|
84
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Create database-specific date literals
|
85
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_date_literal'>date_literal</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>2025-01-01</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span>
|
86
|
+
<span class='comment'># PostgreSQL: '2025-01-01'::DATE
|
87
|
+
</span><span class='comment'># SQL Server: '2025-01-01'
|
88
|
+
</span><span class='comment'># MySQL: '2025-01-01'
|
89
|
+
</span>
|
90
|
+
<span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_date_time_literal'>date_time_literal</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>2025-01-01 10:30:00</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span>
|
91
|
+
<span class='comment'># PostgreSQL: '2025-01-01 10:30:00'::TIMESTAMP
|
92
|
+
</span><span class='comment'># SQL Server: '2025-01-01 10:30:00'
|
93
|
+
</span></code></pre>
|
96
94
|
|
97
95
|
<h4 id="date-arithmetic">Date Arithmetic</h4>
|
98
96
|
|
99
|
-
<
|
100
|
-
|
101
|
-
|
102
|
-
|
103
|
-
|
104
|
-
|
105
|
-
|
106
|
-
<
|
107
|
-
|
108
|
-
```</p>
|
97
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Add/subtract time periods
|
98
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_date_add'>date_add</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>created_at</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='int'>30</span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>day</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span>
|
99
|
+
<span class='comment'># PostgreSQL: (created_at + '30 day'::interval)
|
100
|
+
</span><span class='comment'># SQL Server: DATEADD(day, 30, created_at)
|
101
|
+
</span><span class='comment'># MySQL: TIMESTAMPADD(day, 30, created_at)
|
102
|
+
</span>
|
103
|
+
<span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_date_diff'>date_diff</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>end_date</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>start_date</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>day</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span>
|
104
|
+
<span class='comment'># Calculate difference between dates in specified units
|
105
|
+
</span></code></pre>
|
109
106
|
|
110
107
|
<h4 id="date-extraction">Date Extraction</h4>
|
111
108
|
|
112
|
-
<
|
113
|
-
|
114
|
-
|
115
|
-
|
116
|
-
|
117
|
-
|
118
|
-
<
|
119
|
-
adapter
|
120
|
-
|
121
|
-
```</p>
|
109
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Extract date parts
|
110
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_extract_year'>extract_year</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>created_at</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span>
|
111
|
+
<span class='comment'># PostgreSQL: extract(year from created_at)
|
112
|
+
</span><span class='comment'># SQL Server: DATEPART(year, created_at)
|
113
|
+
</span>
|
114
|
+
<span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_extract_month'>extract_month</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>created_at</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span>
|
115
|
+
<span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_extract_day_of_week'>extract_day_of_week</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>created_at</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span>
|
116
|
+
<span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_extract_quarter'>extract_quarter</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>created_at</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span>
|
117
|
+
</code></pre>
|
122
118
|
|
123
119
|
<h3 id="string-functions">String Functions</h3>
|
124
120
|
|
125
|
-
<
|
126
|
-
|
127
|
-
adapter
|
128
|
-
adapter
|
129
|
-
|
130
|
-
|
131
|
-
|
132
|
-
|
133
|
-
|
134
|
-
```</p>
|
121
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># String manipulation
|
122
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_trim'>trim</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>column_name</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span> <span class='comment'># Remove whitespace
|
123
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_upper_case'>upper_case</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>column_name</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span> <span class='comment'># Convert to uppercase
|
124
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_lower_case'>lower_case</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>column_name</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span> <span class='comment'># Convert to lowercase
|
125
|
+
</span>
|
126
|
+
<span class='comment'># Quoting and literals
|
127
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_quote'>quote</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>column_name</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span> <span class='comment'># Database-specific column quoting
|
128
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_string_literal'>string_literal</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>value</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span> <span class='comment'># Database-specific string literals
|
129
|
+
</span></code></pre>
|
135
130
|
|
136
131
|
<h3 id="null-handling">Null Handling</h3>
|
137
132
|
|
138
|
-
<
|
139
|
-
|
140
|
-
|
141
|
-
|
142
|
-
|
143
|
-
|
144
|
-
<
|
145
|
-
|
146
|
-
|
147
|
-
<
|
148
|
-
|
149
|
-
```</p>
|
133
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Null value handling
|
134
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_if_null'>if_null</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>column1</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>'default'</span><span class='tstring_end'>"</span></span><span class='rparen'>)</span>
|
135
|
+
<span class='comment'># PostgreSQL: COALESCE(column1, 'default')
|
136
|
+
</span><span class='comment'># SQL Server: ISNULL(column1, 'default')
|
137
|
+
</span>
|
138
|
+
<span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_null_if'>null_if</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>column1</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>'empty'</span><span class='tstring_end'>"</span></span><span class='rparen'>)</span>
|
139
|
+
<span class='comment'># Returns NULL if column1 equals 'empty'
|
140
|
+
</span>
|
141
|
+
<span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_null_if_zero'>null_if_zero</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>numeric_column</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span>
|
142
|
+
<span class='comment'># Returns NULL if numeric_column equals 0
|
143
|
+
</span></code></pre>
|
150
144
|
|
151
145
|
<h3 id="array-functions">Array Functions</h3>
|
152
146
|
|
153
147
|
<p>Available for databases that support array operations (PostgreSQL, Druid):</p>
|
154
148
|
|
155
|
-
<
|
156
|
-
|
157
|
-
|
158
|
-
|
159
|
-
|
149
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Check if array contains any values from a list
|
150
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_array_in_list'>array_in_list</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>tags</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>'tech', 'science'</span><span class='tstring_end'>"</span></span><span class='rparen'>)</span>
|
151
|
+
<span class='comment'># PostgreSQL: tags && ARRAY['tech', 'science']
|
152
|
+
</span><span class='comment'># Druid: MV_OVERLAP(tags, ARRAY['tech', 'science'])
|
153
|
+
</span>
|
154
|
+
<span class='comment'># Check if array excludes all values from a list
|
155
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_array_exclude_list'>array_exclude_list</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>categories</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>'spam', 'test'</span><span class='tstring_end'>"</span></span><span class='rparen'>)</span>
|
160
156
|
|
161
|
-
<
|
162
|
-
|
163
|
-
|
164
|
-
|
165
|
-
|
166
|
-
# PostgreSQL: CROSS JOIN UNNEST(tags) AS tag_alias
|
167
|
-
# Druid: CROSS JOIN UNNEST(MV_TO_ARRAY(tags)) tag_alias
|
168
|
-
```</p>
|
157
|
+
<span class='comment'># Unnest/explode array for joins
|
158
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_array_unnest_join'>array_unnest_join</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>tags</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>tag_alias</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span>
|
159
|
+
<span class='comment'># PostgreSQL: CROSS JOIN UNNEST(tags) AS tag_alias
|
160
|
+
</span><span class='comment'># Druid: CROSS JOIN UNNEST(MV_TO_ARRAY(tags)) tag_alias
|
161
|
+
</span></code></pre>
|
169
162
|
|
170
163
|
<h3 id="type-casting">Type Casting</h3>
|
171
164
|
|
172
|
-
<
|
173
|
-
|
174
|
-
|
175
|
-
|
176
|
-
|
177
|
-
|
178
|
-
</code></p>
|
165
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Database-specific type casting
|
166
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_cast'>cast</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>column_name</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>INTEGER</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span>
|
167
|
+
<span class='comment'># PostgreSQL: column_name::INTEGER
|
168
|
+
</span><span class='comment'># SQL Server: CAST(column_name AS INTEGER)
|
169
|
+
</span><span class='comment'># MySQL: CAST(column_name AS SIGNED)
|
170
|
+
</span></code></pre>
|
179
171
|
|
180
172
|
<h2 id="streaming-and-large-result-sets">Streaming and Large Result Sets</h2>
|
181
173
|
|
182
174
|
<h3 id="basic-streaming">Basic Streaming</h3>
|
183
175
|
|
184
|
-
<
|
185
|
-
|
186
|
-
|
187
|
-
|
188
|
-
end</p>
|
176
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Stream results directly to a file
|
177
|
+
</span><span class='const'>File</span><span class='period'>.</span><span class='id identifier rubyid_open'>open</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>large_export.csv</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>w</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_file'>file</span><span class='op'>|</span>
|
178
|
+
<span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_execute_stream'>execute_stream</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SELECT * FROM large_table</span><span class='tstring_end'>"</span></span><span class='comma'>,</span> <span class='id identifier rubyid_file'>file</span><span class='rparen'>)</span>
|
179
|
+
<span class='kw'>end</span>
|
189
180
|
|
190
|
-
<
|
191
|
-
|
192
|
-
|
193
|
-
process_data_chunk(chunk)
|
194
|
-
end
|
195
|
-
|
181
|
+
<span class='comment'># Stream with custom processing
|
182
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_stream'>stream</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SELECT * FROM large_table</span><span class='tstring_end'>"</span></span><span class='rparen'>)</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_chunk'>chunk</span><span class='op'>|</span>
|
183
|
+
<span class='comment'># Process each chunk as it arrives
|
184
|
+
</span> <span class='id identifier rubyid_process_data_chunk'>process_data_chunk</span><span class='lparen'>(</span><span class='id identifier rubyid_chunk'>chunk</span><span class='rparen'>)</span>
|
185
|
+
<span class='kw'>end</span>
|
186
|
+
</code></pre>
|
196
187
|
|
197
188
|
<h3 id="streaming-with-statistics">Streaming with Statistics</h3>
|
198
189
|
|
199
|
-
<
|
200
|
-
|
201
|
-
|
202
|
-
|
203
|
-
|
204
|
-
<
|
205
|
-
|
206
|
-
|
207
|
-
<
|
208
|
-
|
209
|
-
|
210
|
-
puts
|
211
|
-
puts
|
212
|
-
|
213
|
-
|
214
|
-
|
215
|
-
|
216
|
-
|
217
|
-
|
218
|
-
|
219
|
-
<
|
220
|
-
<
|
221
|
-
|
222
|
-
```</p>
|
190
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Create streaming stats collector
|
191
|
+
</span><span class='id identifier rubyid_stats'>stats</span> <span class='op'>=</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="DWH/StreamingStats.html" title="DWH::StreamingStats (class)">StreamingStats</a></span></span><span class='period'>.</span><span class='id identifier rubyid_new'><span class='object_link'><a href="DWH/StreamingStats.html#initialize-instance_method" title="DWH::StreamingStats#initialize (method)">new</a></span></span><span class='lparen'>(</span><span class='int'>10_000</span><span class='rparen'>)</span> <span class='comment'># Keep 10k rows in memory for preview
|
192
|
+
</span>
|
193
|
+
<span class='comment'># Stream with stats tracking
|
194
|
+
</span><span class='const'>File</span><span class='period'>.</span><span class='id identifier rubyid_open'>open</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>export.csv</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>w</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_file'>file</span><span class='op'>|</span>
|
195
|
+
<span class='id identifier rubyid_exec_thread'>exec_thread</span> <span class='op'>=</span> <span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_execute_stream'>execute_stream</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SELECT * FROM large_table</span><span class='tstring_end'>"</span></span><span class='comma'>,</span> <span class='id identifier rubyid_file'>file</span><span class='comma'>,</span> <span class='label'>stats:</span> <span class='id identifier rubyid_stats'>stats</span><span class='rparen'>)</span>
|
196
|
+
|
197
|
+
<span class='comment'># Monitor progress in another thread
|
198
|
+
</span> <span class='const'>Thread</span><span class='period'>.</span><span class='id identifier rubyid_new'>new</span> <span class='kw'>do</span>
|
199
|
+
<span class='id identifier rubyid_loop'>loop</span> <span class='kw'>do</span>
|
200
|
+
<span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Processed: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_stats'>stats</span><span class='period'>.</span><span class='id identifier rubyid_total_rows'>total_rows</span><span class='embexpr_end'>}</span><span class='tstring_content'> rows</span><span class='tstring_end'>"</span></span>
|
201
|
+
<span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Preview size: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_stats'>stats</span><span class='period'>.</span><span class='id identifier rubyid_data'>data</span><span class='period'>.</span><span class='id identifier rubyid_size'>size</span><span class='embexpr_end'>}</span><span class='tstring_content'> rows</span><span class='tstring_end'>"</span></span>
|
202
|
+
<span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Max row size: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_stats'>stats</span><span class='period'>.</span><span class='id identifier rubyid_max_row_size'>max_row_size</span><span class='embexpr_end'>}</span><span class='tstring_content'> bytes</span><span class='tstring_end'>"</span></span>
|
203
|
+
<span class='id identifier rubyid_sleep'>sleep</span><span class='lparen'>(</span><span class='int'>5</span><span class='rparen'>)</span>
|
204
|
+
<span class='kw'>break</span> <span class='kw'>unless</span> <span class='id identifier rubyid_exec_thread'>exec_thread</span><span class='period'>.</span><span class='id identifier rubyid_alive?'>alive?</span>
|
205
|
+
<span class='kw'>end</span>
|
206
|
+
<span class='kw'>end</span>
|
207
|
+
<span class='kw'>end</span>
|
208
|
+
|
209
|
+
<span class='comment'># Access collected statistics
|
210
|
+
</span><span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Final count: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_stats'>stats</span><span class='period'>.</span><span class='id identifier rubyid_total_rows'>total_rows</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
211
|
+
<span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Sample data: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_stats'>stats</span><span class='period'>.</span><span class='id identifier rubyid_data'>data</span><span class='period'>.</span><span class='id identifier rubyid_first'>first</span><span class='lparen'>(</span><span class='int'>5</span><span class='rparen'>)</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
212
|
+
</code></pre>
|
223
213
|
|
224
214
|
<h3 id="memory-management">Memory Management</h3>
|
225
215
|
|
226
|
-
<
|
227
|
-
|
228
|
-
|
229
|
-
|
230
|
-
|
231
|
-
<p>stats.reset</p>
|
216
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Configure streaming stats memory usage
|
217
|
+
</span><span class='id identifier rubyid_stats'>stats</span> <span class='op'>=</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="DWH/StreamingStats.html" title="DWH::StreamingStats (class)">StreamingStats</a></span></span><span class='period'>.</span><span class='id identifier rubyid_new'><span class='object_link'><a href="DWH/StreamingStats.html#initialize-instance_method" title="DWH::StreamingStats#initialize (method)">new</a></span></span><span class='lparen'>(</span><span class='int'>50_000</span><span class='rparen'>)</span> <span class='comment'># Keep more data for larger previews
|
218
|
+
</span>
|
219
|
+
<span class='comment'># Reset stats for reuse
|
220
|
+
</span><span class='id identifier rubyid_stats'>stats</span><span class='period'>.</span><span class='id identifier rubyid_reset'>reset</span>
|
232
221
|
|
233
|
-
<
|
234
|
-
|
235
|
-
current_data = stats
|
236
|
-
|
222
|
+
<span class='comment'># Manual memory management
|
223
|
+
</span><span class='id identifier rubyid_stats'>stats</span><span class='period'>.</span><span class='id identifier rubyid_add_row'>add_row</span><span class='lparen'>(</span><span class='lbracket'>[</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>col1</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>col2</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>col3</span><span class='tstring_end'>'</span></span><span class='rbracket'>]</span><span class='rparen'>)</span>
|
224
|
+
<span class='id identifier rubyid_current_data'>current_data</span> <span class='op'>=</span> <span class='id identifier rubyid_stats'>stats</span><span class='period'>.</span><span class='id identifier rubyid_data'>data</span> <span class='comment'># Thread-safe access
|
225
|
+
</span></code></pre>
|
237
226
|
|
238
227
|
<h2 id="connection-pooling">Connection Pooling</h2>
|
239
228
|
|
240
229
|
<h3 id="creating-connection-pools">Creating Connection Pools</h3>
|
241
230
|
|
242
|
-
<
|
243
|
-
#
|
244
|
-
|
245
|
-
|
246
|
-
|
247
|
-
|
248
|
-
|
249
|
-
}, size: 10, timeout: 5)</p>
|
231
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Create a named connection pool
|
232
|
+
</span><span class='id identifier rubyid_pool'>pool</span> <span class='op'>=</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='period'>.</span><span class='id identifier rubyid_pool'><span class='object_link'><a href="DWH/Factory.html#pool-instance_method" title="DWH::Factory#pool (method)">pool</a></span></span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>analytics_pool</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='symbol'>:postgres</span><span class='comma'>,</span> <span class='lbrace'>{</span>
|
233
|
+
<span class='label'>host:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>localhost</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
234
|
+
<span class='label'>database:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>analytics</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
235
|
+
<span class='label'>username:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>analyst</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
236
|
+
<span class='label'>password:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>password</span><span class='tstring_end'>'</span></span>
|
237
|
+
<span class='rbrace'>}</span><span class='comma'>,</span> <span class='label'>size:</span> <span class='int'>10</span><span class='comma'>,</span> <span class='label'>timeout:</span> <span class='int'>5</span><span class='rparen'>)</span>
|
250
238
|
|
251
|
-
<
|
252
|
-
|
253
|
-
reporting_pool = DWH.pool(
|
254
|
-
|
239
|
+
<span class='comment'># Multiple pools for different databases
|
240
|
+
</span><span class='id identifier rubyid_etl_pool'>etl_pool</span> <span class='op'>=</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='period'>.</span><span class='id identifier rubyid_pool'><span class='object_link'><a href="DWH/Factory.html#pool-instance_method" title="DWH::Factory#pool (method)">pool</a></span></span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>etl_pool</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='symbol'>:postgres</span><span class='comma'>,</span> <span class='id identifier rubyid_etl_config'>etl_config</span><span class='comma'>,</span> <span class='label'>size:</span> <span class='int'>5</span><span class='rparen'>)</span>
|
241
|
+
<span class='id identifier rubyid_reporting_pool'>reporting_pool</span> <span class='op'>=</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='period'>.</span><span class='id identifier rubyid_pool'><span class='object_link'><a href="DWH/Factory.html#pool-instance_method" title="DWH::Factory#pool (method)">pool</a></span></span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>reporting_pool</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='symbol'>:mysql</span><span class='comma'>,</span> <span class='id identifier rubyid_reporting_config'>reporting_config</span><span class='comma'>,</span> <span class='label'>size:</span> <span class='int'>15</span><span class='rparen'>)</span>
|
242
|
+
</code></pre>
|
255
243
|
|
256
244
|
<h3 id="using-connection-pools">Using Connection Pools</h3>
|
257
245
|
|
258
|
-
<
|
259
|
-
|
260
|
-
|
261
|
-
|
262
|
-
|
263
|
-
end</p>
|
246
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Basic pool usage
|
247
|
+
</span><span class='id identifier rubyid_pool'>pool</span><span class='period'>.</span><span class='id identifier rubyid_with'>with</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_connection'>connection</span><span class='op'>|</span>
|
248
|
+
<span class='id identifier rubyid_results'>results</span> <span class='op'>=</span> <span class='id identifier rubyid_connection'>connection</span><span class='period'>.</span><span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SELECT COUNT(*) FROM users</span><span class='tstring_end'>"</span></span><span class='rparen'>)</span>
|
249
|
+
<span class='id identifier rubyid_metadata'>metadata</span> <span class='op'>=</span> <span class='id identifier rubyid_connection'>connection</span><span class='period'>.</span><span class='id identifier rubyid_metadata'>metadata</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>orders</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span>
|
250
|
+
<span class='kw'>end</span>
|
264
251
|
|
265
|
-
<
|
266
|
-
|
267
|
-
users = conn1
|
252
|
+
<span class='comment'># Nested pool operations
|
253
|
+
</span><span class='id identifier rubyid_pool'>pool</span><span class='period'>.</span><span class='id identifier rubyid_with'>with</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_conn1'>conn1</span><span class='op'>|</span>
|
254
|
+
<span class='id identifier rubyid_users'>users</span> <span class='op'>=</span> <span class='id identifier rubyid_conn1'>conn1</span><span class='period'>.</span><span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SELECT id FROM users LIMIT 100</span><span class='tstring_end'>"</span></span><span class='rparen'>)</span>
|
268
255
|
|
269
|
-
<
|
270
|
-
orders = conn2
|
271
|
-
users
|
272
|
-
end
|
273
|
-
end
|
274
|
-
|
256
|
+
<span class='id identifier rubyid_pool'>pool</span><span class='period'>.</span><span class='id identifier rubyid_with'>with</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_conn2'>conn2</span><span class='op'>|</span> <span class='comment'># Gets different connection from pool
|
257
|
+
</span> <span class='id identifier rubyid_orders'>orders</span> <span class='op'>=</span> <span class='id identifier rubyid_conn2'>conn2</span><span class='period'>.</span><span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SELECT * FROM orders WHERE user_id IN (?)</span><span class='tstring_end'>"</span></span><span class='comma'>,</span>
|
258
|
+
<span class='id identifier rubyid_users'>users</span><span class='period'>.</span><span class='id identifier rubyid_map'>map</span><span class='lparen'>(</span><span class='op'>&</span><span class='symbol'>:first</span><span class='rparen'>)</span><span class='rparen'>)</span>
|
259
|
+
<span class='kw'>end</span>
|
260
|
+
<span class='kw'>end</span>
|
261
|
+
</code></pre>
|
275
262
|
|
276
263
|
<h3 id="pool-management">Pool Management</h3>
|
277
264
|
|
278
|
-
<
|
279
|
-
|
280
|
-
puts
|
281
|
-
puts
|
282
|
-
puts “Active connections: #poolpool.in_use”</p>
|
265
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Check pool status
|
266
|
+
</span><span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Pool size: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_pool'>pool</span><span class='period'>.</span><span class='id identifier rubyid_size'>size</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
267
|
+
<span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Available connections: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_pool'>pool</span><span class='period'>.</span><span class='id identifier rubyid_available'>available</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
268
|
+
<span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Active connections: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_pool'>pool</span><span class='period'>.</span><span class='id identifier rubyid_in_use'>in_use</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
283
269
|
|
284
|
-
<
|
285
|
-
|
270
|
+
<span class='comment'># Graceful shutdown
|
271
|
+
</span><span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='period'>.</span><span class='id identifier rubyid_shutdown'><span class='object_link'><a href="DWH/Factory.html#shutdown-instance_method" title="DWH::Factory#shutdown (method)">shutdown</a></span></span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>analytics_pool</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span>
|
286
272
|
|
287
|
-
<
|
288
|
-
|
289
|
-
|
273
|
+
<span class='comment'># Shutdown all pools
|
274
|
+
</span><span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='period'>.</span><span class='id identifier rubyid_shutdown_all'>shutdown_all</span>
|
275
|
+
</code></pre>
|
290
276
|
|
291
277
|
<h2 id="database-capabilities-detection">Database Capabilities Detection</h2>
|
292
278
|
|
293
279
|
<h3 id="checking-capabilities">Checking Capabilities</h3>
|
294
280
|
|
295
|
-
<
|
296
|
-
|
297
|
-
|
298
|
-
|
299
|
-
|
300
|
-
|
301
|
-
</code></p>
|
281
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Check what features are supported
|
282
|
+
</span><span class='kw'>if</span> <span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_supports_window_functions?'>supports_window_functions?</span>
|
283
|
+
<span class='id identifier rubyid_query'>query</span> <span class='op'>=</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SELECT name, ROW_NUMBER() OVER (ORDER BY created_at) FROM users</span><span class='tstring_end'>"</span></span>
|
284
|
+
<span class='id identifier rubyid_results'>results</span> <span class='op'>=</span> <span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='id identifier rubyid_query'>query</span><span class='rparen'>)</span>
|
285
|
+
<span class='kw'>end</span>
|
286
|
+
</code></pre>
|
302
287
|
|
303
288
|
<h3 id="available-capability-checks">Available Capability Checks</h3>
|
304
289
|
|
305
290
|
<ul>
|
306
|
-
|
307
|
-
|
308
|
-
|
309
|
-
|
310
|
-
|
311
|
-
|
312
|
-
|
313
|
-
|
291
|
+
<li><code>supports_table_join?</code> - Basic JOIN support</li>
|
292
|
+
<li><code>supports_full_join?</code> - FULL OUTER JOIN support</li>
|
293
|
+
<li><code>supports_cross_join?</code> - CROSS JOIN support</li>
|
294
|
+
<li><code>supports_sub_queries?</code> - Subquery support</li>
|
295
|
+
<li><code>supports_common_table_expressions?</code> - CTE support</li>
|
296
|
+
<li><code>supports_temp_tables?</code> - Temporary table support</li>
|
297
|
+
<li><code>supports_window_functions?</code> - Window function support</li>
|
298
|
+
<li><code>supports_array_functions?</code> - Array operation support</li>
|
314
299
|
</ul>
|
315
300
|
|
316
301
|
<h2 id="performance-optimization">Performance Optimization</h2>
|
317
302
|
|
318
303
|
<h3 id="query-timeouts">Query Timeouts</h3>
|
319
304
|
|
320
|
-
<
|
321
|
-
#
|
322
|
-
|
323
|
-
|
324
|
-
|
325
|
-
|
326
|
-
|
327
|
-
|
328
|
-
|
329
|
-
<
|
330
|
-
<
|
331
|
-
|
332
|
-
|
333
|
-
|
334
|
-
|
335
|
-
|
336
|
-
```</p>
|
305
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Set query timeouts per adapter
|
306
|
+
</span><span class='id identifier rubyid_postgres'>postgres</span> <span class='op'>=</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='period'>.</span><span class='id identifier rubyid_create'><span class='object_link'><a href="DWH/Factory.html#create-instance_method" title="DWH::Factory#create (method)">create</a></span></span><span class='lparen'>(</span><span class='symbol'>:postgres</span><span class='comma'>,</span> <span class='lbrace'>{</span>
|
307
|
+
<span class='label'>host:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>localhost</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
308
|
+
<span class='label'>database:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>mydb</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
309
|
+
<span class='label'>username:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>user</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
310
|
+
<span class='label'>query_timeout:</span> <span class='int'>1800</span> <span class='comment'># 30 minutes
|
311
|
+
</span><span class='rbrace'>}</span><span class='rparen'>)</span>
|
312
|
+
|
313
|
+
<span class='comment'># For long-running analytical queries
|
314
|
+
</span><span class='id identifier rubyid_druid'>druid</span> <span class='op'>=</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='period'>.</span><span class='id identifier rubyid_create'><span class='object_link'><a href="DWH/Factory.html#create-instance_method" title="DWH::Factory#create (method)">create</a></span></span><span class='lparen'>(</span><span class='symbol'>:druid</span><span class='comma'>,</span> <span class='lbrace'>{</span>
|
315
|
+
<span class='label'>host:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>localhost</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
316
|
+
<span class='label'>port:</span> <span class='int'>8080</span><span class='comma'>,</span>
|
317
|
+
<span class='label'>protocol:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>http</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
318
|
+
<span class='label'>query_timeout:</span> <span class='int'>3600</span> <span class='comment'># 1 hour
|
319
|
+
</span><span class='rbrace'>}</span><span class='rparen'>)</span>
|
320
|
+
</code></pre>
|
337
321
|
|
338
322
|
<h3 id="result-format-optimization">Result Format Optimization</h3>
|
339
323
|
|
340
|
-
<
|
341
|
-
|
342
|
-
|
343
|
-
|
344
|
-
|
345
|
-
|
346
|
-
</code></p>
|
324
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Choose appropriate result format for your use case
|
325
|
+
</span><span class='id identifier rubyid_arrays'>arrays</span> <span class='op'>=</span> <span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='id identifier rubyid_sql'>sql</span><span class='comma'>,</span> <span class='label'>format:</span> <span class='symbol'>:array</span><span class='rparen'>)</span> <span class='comment'># Fastest, least memory
|
326
|
+
</span><span class='id identifier rubyid_objects'>objects</span> <span class='op'>=</span> <span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='id identifier rubyid_sql'>sql</span><span class='comma'>,</span> <span class='label'>format:</span> <span class='symbol'>:object</span><span class='rparen'>)</span> <span class='comment'># Hash access, more memory
|
327
|
+
</span><span class='id identifier rubyid_csv'>csv</span> <span class='op'>=</span> <span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='id identifier rubyid_sql'>sql</span><span class='comma'>,</span> <span class='label'>format:</span> <span class='symbol'>:csv</span><span class='rparen'>)</span> <span class='comment'># String format
|
328
|
+
</span><span class='id identifier rubyid_native'>native</span> <span class='op'>=</span> <span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='id identifier rubyid_sql'>sql</span><span class='comma'>,</span> <span class='label'>format:</span> <span class='symbol'>:native</span><span class='rparen'>)</span> <span class='comment'># Database's native format
|
329
|
+
</span></code></pre>
|
347
330
|
|
348
331
|
<h3 id="streaming-for-large-results">Streaming for Large Results</h3>
|
349
332
|
|
350
|
-
<
|
351
|
-
|
352
|
-
|
353
|
-
|
354
|
-
|
355
|
-
<
|
356
|
-
|
357
|
-
|
358
|
-
|
359
|
-
|
360
|
-
<
|
361
|
-
<
|
362
|
-
|
363
|
-
|
364
|
-
|
365
|
-
|
366
|
-
|
367
|
-
|
368
|
-
```</p>
|
333
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Use streaming for large result sets
|
334
|
+
</span><span class='kw'>def</span> <span class='id identifier rubyid_export_large_table'>export_large_table</span><span class='lparen'>(</span><span class='id identifier rubyid_adapter'>adapter</span><span class='comma'>,</span> <span class='id identifier rubyid_table_name'>table_name</span><span class='comma'>,</span> <span class='id identifier rubyid_output_file'>output_file</span><span class='rparen'>)</span>
|
335
|
+
<span class='id identifier rubyid_query'>query</span> <span class='op'>=</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SELECT * FROM </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_table_name'>table_name</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
336
|
+
|
337
|
+
<span class='const'>File</span><span class='period'>.</span><span class='id identifier rubyid_open'>open</span><span class='lparen'>(</span><span class='id identifier rubyid_output_file'>output_file</span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>w</span><span class='tstring_end'>'</span></span><span class='rparen'>)</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_file'>file</span><span class='op'>|</span>
|
338
|
+
<span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_execute_stream'>execute_stream</span><span class='lparen'>(</span><span class='id identifier rubyid_query'>query</span><span class='comma'>,</span> <span class='id identifier rubyid_file'>file</span><span class='rparen'>)</span>
|
339
|
+
<span class='kw'>end</span>
|
340
|
+
<span class='kw'>end</span>
|
341
|
+
|
342
|
+
<span class='comment'># Chunk processing for memory efficiency
|
343
|
+
</span><span class='kw'>def</span> <span class='id identifier rubyid_process_large_dataset'>process_large_dataset</span><span class='lparen'>(</span><span class='id identifier rubyid_adapter'>adapter</span><span class='comma'>,</span> <span class='id identifier rubyid_query'>query</span><span class='rparen'>)</span>
|
344
|
+
<span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_stream'>stream</span><span class='lparen'>(</span><span class='id identifier rubyid_query'>query</span><span class='rparen'>)</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_chunk'>chunk</span><span class='op'>|</span>
|
345
|
+
<span class='comment'># Process each chunk immediately
|
346
|
+
</span> <span class='comment'># Avoids loading entire result set into memory
|
347
|
+
</span> <span class='id identifier rubyid_chunk'>chunk</span><span class='period'>.</span><span class='id identifier rubyid_each'>each</span> <span class='lbrace'>{</span> <span class='op'>|</span><span class='id identifier rubyid_row'>row</span><span class='op'>|</span> <span class='id identifier rubyid_process_row'>process_row</span><span class='lparen'>(</span><span class='id identifier rubyid_row'>row</span><span class='rparen'>)</span> <span class='rbrace'>}</span>
|
348
|
+
<span class='kw'>end</span>
|
349
|
+
<span class='kw'>end</span>
|
350
|
+
</code></pre>
|
369
351
|
|
370
352
|
<h2 id="error-handling-and-debugging">Error Handling and Debugging</h2>
|
371
353
|
|
372
354
|
<h3 id="comprehensive-error-handling">Comprehensive Error Handling</h3>
|
373
355
|
|
374
|
-
<
|
375
|
-
|
376
|
-
|
377
|
-
|
378
|
-
|
379
|
-
puts
|
380
|
-
|
381
|
-
|
382
|
-
|
383
|
-
|
384
|
-
|
385
|
-
|
386
|
-
|
387
|
-
|
388
|
-
|
389
|
-
|
390
|
-
|
391
|
-
|
392
|
-
</code></p>
|
356
|
+
<pre class="code ruby"><code class="ruby"><span class='kw'>begin</span>
|
357
|
+
<span class='id identifier rubyid_results'>results</span> <span class='op'>=</span> <span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_execute'>execute</span><span class='lparen'>(</span><span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SELECT * FROM table</span><span class='tstring_end'>"</span></span><span class='rparen'>)</span>
|
358
|
+
<span class='kw'>rescue</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="DWH/ExecutionError.html" title="DWH::ExecutionError (class)">ExecutionError</a></span></span> <span class='op'>=></span> <span class='id identifier rubyid_e'>e</span>
|
359
|
+
<span class='comment'># Query execution failed
|
360
|
+
</span> <span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Query failed: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_e'>e</span><span class='period'>.</span><span class='id identifier rubyid_message'>message</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
361
|
+
<span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>SQL: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_e'>e</span><span class='period'>.</span><span class='id identifier rubyid_sql'>sql</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span> <span class='kw'>if</span> <span class='id identifier rubyid_e'>e</span><span class='period'>.</span><span class='id identifier rubyid_respond_to?'>respond_to?</span><span class='lparen'>(</span><span class='symbol'>:sql</span><span class='rparen'>)</span>
|
362
|
+
<span class='kw'>rescue</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="DWH/ConnectionError.html" title="DWH::ConnectionError (class)">ConnectionError</a></span></span> <span class='op'>=></span> <span class='id identifier rubyid_e'>e</span>
|
363
|
+
<span class='comment'># Connection issues
|
364
|
+
</span> <span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Connection failed: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_e'>e</span><span class='period'>.</span><span class='id identifier rubyid_message'>message</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
365
|
+
<span class='comment'># Implement retry logic
|
366
|
+
</span><span class='kw'>rescue</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="DWH/ConfigError.html" title="DWH::ConfigError (class)">ConfigError</a></span></span> <span class='op'>=></span> <span class='id identifier rubyid_e'>e</span>
|
367
|
+
<span class='comment'># Configuration problems
|
368
|
+
</span> <span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Configuration error: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_e'>e</span><span class='period'>.</span><span class='id identifier rubyid_message'>message</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
369
|
+
<span class='kw'>rescue</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='op'>::</span><span class='const'><span class='object_link'><a href="DWH/UnsupportedCapability.html" title="DWH::UnsupportedCapability (class)">UnsupportedCapability</a></span></span> <span class='op'>=></span> <span class='id identifier rubyid_e'>e</span>
|
370
|
+
<span class='comment'># Attempted unsupported operation
|
371
|
+
</span> <span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>Feature not supported: </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_e'>e</span><span class='period'>.</span><span class='id identifier rubyid_message'>message</span><span class='embexpr_end'>}</span><span class='tstring_end'>"</span></span>
|
372
|
+
<span class='kw'>end</span>
|
373
|
+
</code></pre>
|
393
374
|
|
394
375
|
<h2 id="custom-settings-and-overrides">Custom Settings and Overrides</h2>
|
395
376
|
|
396
377
|
<h3 id="runtime-settings-modification">Runtime Settings Modification</h3>
|
397
378
|
|
398
|
-
<
|
399
|
-
|
400
|
-
|
401
|
-
|
402
|
-
|
403
|
-
|
404
|
-
|
405
|
-
|
406
|
-
|
407
|
-
|
408
|
-
|
409
|
-
|
410
|
-
|
411
|
-
|
412
|
-
<
|
413
|
-
|
414
|
-
|
415
|
-
|
416
|
-
|
417
|
-
|
418
|
-
|
419
|
-
```</p>
|
379
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Create adapter with custom settings
|
380
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span> <span class='op'>=</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='period'>.</span><span class='id identifier rubyid_create'><span class='object_link'><a href="DWH/Factory.html#create-instance_method" title="DWH::Factory#create (method)">create</a></span></span><span class='lparen'>(</span><span class='symbol'>:postgres</span><span class='comma'>,</span> <span class='lbrace'>{</span>
|
381
|
+
<span class='label'>host:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>localhost</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
382
|
+
<span class='label'>database:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>mydb</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
383
|
+
<span class='label'>username:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>user</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
384
|
+
<span class='label'>settings:</span> <span class='lbrace'>{</span>
|
385
|
+
<span class='label'>quote:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>`@exp`</span><span class='tstring_end'>'</span></span><span class='comma'>,</span> <span class='comment'># Use backticks instead of double quotes
|
386
|
+
</span> <span class='label'>supports_window_functions:</span> <span class='kw'>false</span><span class='comma'>,</span> <span class='comment'># Force disable window functions
|
387
|
+
</span> <span class='label'>temp_table_type:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>cte</span><span class='tstring_end'>'</span></span> <span class='comment'># Prefer CTEs over subqueries
|
388
|
+
</span> <span class='rbrace'>}</span>
|
389
|
+
<span class='rbrace'>}</span><span class='rparen'>)</span>
|
390
|
+
|
391
|
+
<span class='comment'># Modify settings at runtime
|
392
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_alter_settings'>alter_settings</span><span class='lparen'>(</span><span class='lbrace'>{</span>
|
393
|
+
<span class='label'>supports_full_join:</span> <span class='kw'>false</span><span class='comma'>,</span> <span class='comment'># Disable FULL JOINs
|
394
|
+
</span> <span class='label'>final_pass_measure_join_type:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>inner</span><span class='tstring_end'>'</span></span> <span class='comment'># Use INNER JOINs
|
395
|
+
</span><span class='rbrace'>}</span><span class='rparen'>)</span>
|
396
|
+
|
397
|
+
<span class='comment'># Reset to original settings
|
398
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span><span class='period'>.</span><span class='id identifier rubyid_reset_settings'>reset_settings</span>
|
399
|
+
</code></pre>
|
420
400
|
|
421
401
|
<h3 id="custom-function-mappings">Custom Function Mappings</h3>
|
422
402
|
|
423
|
-
<
|
424
|
-
#
|
425
|
-
|
426
|
-
|
427
|
-
|
428
|
-
|
429
|
-
|
430
|
-
|
431
|
-
|
432
|
-
|
433
|
-
|
434
|
-
|
435
|
-
</code></p>
|
403
|
+
<pre class="code ruby"><code class="ruby"><span class='comment'># Override specific function translations
|
404
|
+
</span><span class='id identifier rubyid_adapter'>adapter</span> <span class='op'>=</span> <span class='const'><span class='object_link'><a href="DWH.html" title="DWH (module)">DWH</a></span></span><span class='period'>.</span><span class='id identifier rubyid_create'><span class='object_link'><a href="DWH/Factory.html#create-instance_method" title="DWH::Factory#create (method)">create</a></span></span><span class='lparen'>(</span><span class='symbol'>:postgres</span><span class='comma'>,</span> <span class='lbrace'>{</span>
|
405
|
+
<span class='label'>host:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>localhost</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
406
|
+
<span class='label'>database:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>mydb</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
407
|
+
<span class='label'>username:</span> <span class='tstring'><span class='tstring_beg'>'</span><span class='tstring_content'>user</span><span class='tstring_end'>'</span></span><span class='comma'>,</span>
|
408
|
+
<span class='label'>settings:</span> <span class='lbrace'>{</span>
|
409
|
+
<span class='label'>truncate_date:</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>DATE_TRUNC('@unit', @exp)</span><span class='tstring_end'>"</span></span><span class='comma'>,</span> <span class='comment'># Custom date truncation
|
410
|
+
</span> <span class='label'>cast:</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>@exp::@type</span><span class='tstring_end'>"</span></span><span class='comma'>,</span> <span class='comment'># PostgreSQL-style casting
|
411
|
+
</span> <span class='label'>null_if:</span> <span class='tstring'><span class='tstring_beg'>"</span><span class='tstring_content'>CASE WHEN @exp = @target THEN NULL ELSE @exp END</span><span class='tstring_end'>"</span></span> <span class='comment'># Custom NULLIF
|
412
|
+
</span> <span class='rbrace'>}</span>
|
413
|
+
<span class='rbrace'>}</span><span class='rparen'>)</span>
|
414
|
+
</code></pre>
|
436
415
|
</div></div>
|
437
416
|
|
438
417
|
<div id="footer">
|
439
|
-
Generated on
|
418
|
+
Generated on Mon Aug 25 10:59:27 2025 by
|
440
419
|
<a href="https://yardoc.org" title="Yay! A Ruby Documentation Tool" target="_parent">yard</a>
|
441
420
|
0.9.37 (ruby-3.4.4).
|
442
421
|
</div>
|