google-cloud-bigquery 1.12.0 → 1.38.1

Sign up to get free protection for your applications and to get access to all the features.
Files changed (50) hide show
  1. checksums.yaml +4 -4
  2. data/AUTHENTICATION.md +9 -28
  3. data/CHANGELOG.md +372 -1
  4. data/CONTRIBUTING.md +328 -116
  5. data/LOGGING.md +2 -2
  6. data/OVERVIEW.md +21 -20
  7. data/TROUBLESHOOTING.md +2 -8
  8. data/lib/google/cloud/bigquery/argument.rb +197 -0
  9. data/lib/google/cloud/bigquery/convert.rb +154 -170
  10. data/lib/google/cloud/bigquery/copy_job.rb +40 -23
  11. data/lib/google/cloud/bigquery/credentials.rb +5 -12
  12. data/lib/google/cloud/bigquery/data.rb +109 -18
  13. data/lib/google/cloud/bigquery/dataset/access.rb +322 -51
  14. data/lib/google/cloud/bigquery/dataset/list.rb +7 -13
  15. data/lib/google/cloud/bigquery/dataset.rb +960 -279
  16. data/lib/google/cloud/bigquery/external/avro_source.rb +107 -0
  17. data/lib/google/cloud/bigquery/external/bigtable_source/column.rb +404 -0
  18. data/lib/google/cloud/bigquery/external/bigtable_source/column_family.rb +945 -0
  19. data/lib/google/cloud/bigquery/external/bigtable_source.rb +230 -0
  20. data/lib/google/cloud/bigquery/external/csv_source.rb +481 -0
  21. data/lib/google/cloud/bigquery/external/data_source.rb +771 -0
  22. data/lib/google/cloud/bigquery/external/json_source.rb +170 -0
  23. data/lib/google/cloud/bigquery/external/parquet_source.rb +148 -0
  24. data/lib/google/cloud/bigquery/external/sheets_source.rb +166 -0
  25. data/lib/google/cloud/bigquery/external.rb +50 -2256
  26. data/lib/google/cloud/bigquery/extract_job.rb +217 -58
  27. data/lib/google/cloud/bigquery/insert_response.rb +1 -3
  28. data/lib/google/cloud/bigquery/job/list.rb +13 -20
  29. data/lib/google/cloud/bigquery/job.rb +286 -11
  30. data/lib/google/cloud/bigquery/load_job.rb +801 -133
  31. data/lib/google/cloud/bigquery/model/list.rb +5 -9
  32. data/lib/google/cloud/bigquery/model.rb +247 -16
  33. data/lib/google/cloud/bigquery/policy.rb +432 -0
  34. data/lib/google/cloud/bigquery/project/list.rb +6 -11
  35. data/lib/google/cloud/bigquery/project.rb +526 -243
  36. data/lib/google/cloud/bigquery/query_job.rb +584 -125
  37. data/lib/google/cloud/bigquery/routine/list.rb +165 -0
  38. data/lib/google/cloud/bigquery/routine.rb +1227 -0
  39. data/lib/google/cloud/bigquery/schema/field.rb +413 -63
  40. data/lib/google/cloud/bigquery/schema.rb +221 -48
  41. data/lib/google/cloud/bigquery/service.rb +186 -109
  42. data/lib/google/cloud/bigquery/standard_sql.rb +269 -53
  43. data/lib/google/cloud/bigquery/table/async_inserter.rb +86 -42
  44. data/lib/google/cloud/bigquery/table/list.rb +6 -11
  45. data/lib/google/cloud/bigquery/table.rb +1188 -326
  46. data/lib/google/cloud/bigquery/time.rb +6 -0
  47. data/lib/google/cloud/bigquery/version.rb +1 -1
  48. data/lib/google/cloud/bigquery.rb +18 -8
  49. data/lib/google-cloud-bigquery.rb +15 -13
  50. metadata +67 -40
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 4fb1722765075dded454afdeea337efe2c088c118f6f966c55a84ff9bb065df1
4
- data.tar.gz: 2884b2a12b6a5137234dd4a3fd1fc2420ed2c05eaae223ebeae285caa1ba44b8
3
+ metadata.gz: b0548a7d1c7cdbc387d819573b26adca5ea813c3ffda5d9a585f5cdb9e359a59
4
+ data.tar.gz: 42c818041de0fe87d32c9a559dba2c2ee7acfe441fc97fde61ccb6ebf9407c9d
5
5
  SHA512:
6
- metadata.gz: 851560e73e7e1f48b58e08f99ada6f2f822f33f9dfe246e19b6d3abec4b0bfb2ef4802f402e2951ca0e40cb966be2981930fb62608f81d5a57edfa6ae5626d1f
7
- data.tar.gz: b1a0dc302400edfe2a499d0299cc913cf85e4f34e29b460b8f5b3564a33b0e9a13c99d37539cc65afee07b4cbcfd7103bb07b4898866ea68dcab2187105ead1a
6
+ metadata.gz: abb28b6af3f0b30b2b58dbde4a8b0727419d802cf5144b62bd310295bbdffd3a2dc9dd03c11cbabb2792a440a33f54c803b937659c19d4e65973d0e0dc53862f
7
+ data.tar.gz: 832ef3b337a073041d6df5de3f4bb1fab20f1dd6ab1c3dcb20078a710bcc1011fec2f8a574fa9e81b9dc391218115848430ec58acb4897b7bb772cf4614fff86
data/AUTHENTICATION.md CHANGED
@@ -2,7 +2,9 @@
2
2
 
3
3
  In general, the google-cloud-bigquery library uses [Service
4
4
  Account](https://cloud.google.com/iam/docs/creating-managing-service-accounts)
5
- credentials to connect to Google Cloud services. When running on Compute Engine
5
+ credentials to connect to Google Cloud services. When running on Google Cloud
6
+ Platform (GCP), including Google Compute Engine (GCE), Google Kubernetes Engine
7
+ (GKE), Google App Engine (GAE), Google Cloud Functions (GCF) and Cloud Run,
6
8
  the credentials will be discovered automatically. When running on other
7
9
  environments, the Service Account credentials can be specified by providing the
8
10
  path to the [JSON
@@ -35,32 +37,10 @@ providing **Project ID** and **Service Account Credentials** directly in code.
35
37
 
36
38
  ### Google Cloud Platform environments
37
39
 
38
- While running on Google Cloud Platform environments such as Google Compute
39
- Engine, Google App Engine and Google Kubernetes Engine, no extra work is needed.
40
- The **Project ID** and **Credentials** and are discovered automatically. Code
41
- should be written as if already authenticated. Just be sure when you [set up the
42
- GCE instance][gce-how-to], you add the correct scopes for the APIs you want to
43
- access. For example:
44
-
45
- * **All APIs**
46
- * `https://www.googleapis.com/auth/cloud-platform`
47
- * `https://www.googleapis.com/auth/cloud-platform.read-only`
48
- * **BigQuery**
49
- * `https://www.googleapis.com/auth/bigquery`
50
- * `https://www.googleapis.com/auth/bigquery.insertdata`
51
- * **Compute Engine**
52
- * `https://www.googleapis.com/auth/compute`
53
- * **Datastore**
54
- * `https://www.googleapis.com/auth/datastore`
55
- * `https://www.googleapis.com/auth/userinfo.email`
56
- * **DNS**
57
- * `https://www.googleapis.com/auth/ndev.clouddns.readwrite`
58
- * **Pub/Sub**
59
- * `https://www.googleapis.com/auth/pubsub`
60
- * **Storage**
61
- * `https://www.googleapis.com/auth/devstorage.full_control`
62
- * `https://www.googleapis.com/auth/devstorage.read_only`
63
- * `https://www.googleapis.com/auth/devstorage.read_write`
40
+ When running on Google Cloud Platform (GCP), including Google Compute Engine (GCE),
41
+ Google Kubernetes Engine (GKE), Google App Engine (GAE), Google Cloud Functions
42
+ (GCF) and Cloud Run, the **Project ID** and **Credentials** and are discovered
43
+ automatically. Code should be written as if already authenticated.
64
44
 
65
45
  ### Environment Variables
66
46
 
@@ -97,7 +77,8 @@ bigquery = Google::Cloud::Bigquery.new
97
77
 
98
78
  ### Configuration
99
79
 
100
- The **Project ID** and **Credentials JSON** can be configured instead of placing them in environment variables or providing them as arguments.
80
+ The **Project ID** and the path to the **Credentials JSON** file can be configured
81
+ instead of placing them in environment variables or providing them as arguments.
101
82
 
102
83
  ```ruby
103
84
  require "google/cloud/bigquery"
data/CHANGELOG.md CHANGED
@@ -1,5 +1,376 @@
1
1
  # Release History
2
2
 
3
+ ### 1.38.1 / 2022-01-13
4
+
5
+ #### Bug Fixes
6
+
7
+ * Update Bigquery::Data#ddl? to support ALTER TABLE
8
+ * Update Bigquery::QueryJob#ddl? to support ALTER TABLE
9
+ * Remove newline character ('\n') from #ddl? statement types
10
+
11
+ #### Documentation
12
+
13
+ * Update Contributing docs
14
+
15
+ ### 1.38.0 / 2021-11-16
16
+
17
+ #### Features
18
+
19
+ * Add session support
20
+ * Add create_session and session_id params to Project#query_job
21
+ * Add create_session and session_id params to Dataset#query_job
22
+ * Add session_id param to Project#query
23
+ * Add session_id param to Dataset#query
24
+ * Add Job#session_id
25
+ * Add QueryJob::Updater#create_session=
26
+ * Add QueryJob::Updater#session_id=
27
+
28
+ ### 1.37.0 / 2021-10-21
29
+
30
+ #### Features
31
+
32
+ * Add support for Avro options to external data sources
33
+ * Add External::AvroSource
34
+
35
+ ### 1.36.0 / 2021-09-22
36
+
37
+ #### Features
38
+
39
+ * Add Job#delete
40
+
41
+ #### Bug Fixes
42
+
43
+ * Add precision and scale to Field#add_field
44
+
45
+ ### 1.35.1 / 2021-09-14
46
+
47
+ #### Documentation
48
+
49
+ * Wrap character class regex in backticks
50
+
51
+ ### 1.35.0 / 2021-08-12
52
+
53
+ #### Features
54
+
55
+ * Add GEOGRAPHY schema helpers
56
+ * Add LoadJob#geography
57
+ * Add Schema::Field#geography
58
+ * Add Table::Updater#geography
59
+ * Add support for GEOGRAPHY type
60
+ * Add Schema#geography
61
+ * Add support for multistatement transaction statistics in jobs
62
+ * Add Job#transaction_id
63
+
64
+ ### 1.34.0 / 2021-07-20
65
+
66
+ #### Features
67
+
68
+ * Add DmlStatistics
69
+ * Add QueryJob#deleted_row_count
70
+ * Add QueryJob#inserted_row_count
71
+ * Add QueryJob#updated_row_count
72
+ * Add Data#deleted_row_count
73
+ * Add Data#inserted_row_count
74
+ * Add Data#updated_row_count
75
+
76
+ ### 1.33.0 / 2021-07-14
77
+
78
+ #### Features
79
+
80
+ * Add policy tag support (Column ACLs)
81
+ * Add policy_tags to LoadJob field helper methods
82
+ * Add policy_tags to Schema field helper methods
83
+ * Add policy_tags to Schema::Field field helper methods
84
+ * Add policy_tags to Table field helper methods
85
+ * Add Schema::Field#policy_tags
86
+ * Add Schema::Field#policy_tags=
87
+ * Add support for parameterized types
88
+ * Add max_length to LoadJob::Updater#bytes
89
+ * Add max_length to LoadJob::Updater#string
90
+ * Add max_length to Schema#bytes
91
+ * Add max_length to Schema#string
92
+ * Add max_length to Schema::Field#bytes
93
+ * Add max_length to Schema::Field#string
94
+ * Add max_length to Table::Updater#bytes
95
+ * Add max_length to Table::Updater#string
96
+ * Add precision and scale to LoadJob::Updater#bignumeric
97
+ * Add precision and scale to LoadJob::Updater#numeric
98
+ * Add precision and scale to Schema#bignumeric
99
+ * Add precision and scale to Schema#numeric
100
+ * Add precision and scale to Schema::Field#bignumeric
101
+ * Add precision and scale to Schema::Field#numeric
102
+ * Add precision and scale to Table::Updater#bignumeric
103
+ * Add precision and scale to Table::Updater#numeric
104
+ * Add Schema::Field#max_length
105
+ * Add Schema::Field#precision
106
+ * Add Schema::Field#scale
107
+
108
+ ### 1.32.1 / 2021-07-08
109
+
110
+ #### Documentation
111
+
112
+ * Update AUTHENTICATION.md in handwritten packages
113
+
114
+ ### 1.32.0 / 2021-06-21
115
+
116
+ #### Features
117
+
118
+ * Add support for Parquet options
119
+ * feat(bigquery): Add Bigquery::External::ParquetSource
120
+ * Add Parquet options to LoadJob
121
+ * Add LoadJob#parquet_options?
122
+ * Add LoadJob#parquet_enable_list_inference?
123
+ * Add LoadJob#parquet_enum_as_string?
124
+ * Add LoadJob::Updater#parquet_enable_list_inference=
125
+ * Add LoadJob::Updater#parquet_enum_as_string=
126
+
127
+ #### Bug Fixes
128
+
129
+ * Expand googleauth dependency to support future 1.x versions
130
+
131
+ ### 1.31.0 / 2021-04-28
132
+
133
+ #### Features
134
+
135
+ * Add support for mutable clustering configuration
136
+ * Add Table#clustering_fields=
137
+
138
+ ### 1.30.0 / 2021-04-20
139
+
140
+ #### Features
141
+
142
+ * Add support for BIGNUMERIC data type
143
+ * Add support for BIGNUMERIC data insert.
144
+ * Add support for BIGNUMERIC query params.
145
+ * Add #bignumeric to Schema, Table, LoadJob::Updater and Field.
146
+ * Update types tables in docs.
147
+
148
+ #### Bug Fixes
149
+
150
+ * Fix Table#time_partitioning_expiration=
151
+ * Accept nil argument.
152
+
153
+ ### 1.29.0 / 2021-03-10
154
+
155
+ #### Features
156
+
157
+ * Drop support for Ruby 2.4 and add support for Ruby 3.0
158
+
159
+ ### 1.28.0 / 2021-03-09
160
+
161
+ #### Features
162
+
163
+ * Add Materialized View support
164
+ * Add Dataset#create_materialized_view
165
+ * Add Table#materialized_view?
166
+ * Add Table#enable_refresh?
167
+ * Add Table#enable_refresh=
168
+ * Add Table#last_refresh_time
169
+ * Add Table#refresh_interval_ms
170
+ * Add Table#refresh_interval_ms=
171
+
172
+ ### 1.27.0 / 2021-02-10
173
+
174
+ #### Features
175
+
176
+ * Add Job#reservation_usage
177
+ * Add Routine#determinism_level
178
+ * Add Routine#determinism_level
179
+ * Add Routine#determinism_level=
180
+ * Add Routine#determinism_level_deterministic?
181
+ * Add Routine#determinism_level_not_deterministic?
182
+ * Add Routine::Updater#determinism_level=
183
+
184
+ ### 1.26.0 / 2021-01-13
185
+
186
+ #### Features
187
+
188
+ * Add support for Hive Partitioning
189
+ * Add hive partitioning options to External::DataSource
190
+ * Add hive partitioning options to LoadJob and LoadJob::Updater
191
+ * Replace google-api-client with google-apis-bigquery_v2
192
+
193
+ ### 1.25.0 / 2020-11-16
194
+
195
+ #### Features
196
+
197
+ * Add routine (UDF) to Dataset::Access
198
+ * Add support for Table ACLS (IAM Policy)
199
+ * feat(bigquery): Add support for Table ACLS
200
+ * Add Bigquery::Policy
201
+ * Add Table#policy
202
+ * Add Table#test_iam_permissions
203
+ * Add Table#update_policy
204
+
205
+ ### 1.24.0 / 2020-10-29
206
+
207
+ #### Features
208
+
209
+ * Add iamMember to Dataset::Access
210
+
211
+ #### Bug Fixes
212
+
213
+ * Ensure dense encoding of JSON responses
214
+ * Set query param prettyPrint=false for all requests.
215
+ * Upgrade google-api-client to ~> 0.47
216
+
217
+ #### Documentation
218
+
219
+ * Update supported types for time partition type
220
+
221
+ ### 1.23.0 / 2020-09-17
222
+
223
+ #### Features
224
+
225
+ * quota_project can be set via library configuration ([#7627](https://www.github.com/googleapis/google-cloud-ruby/issues/7627))
226
+
227
+ ### 1.22.0 / 2020-09-10
228
+
229
+ #### Features
230
+
231
+ * Add support for ML model export
232
+ * Add model support to Project#extract and #extract_job
233
+ * Add ExtractJob#model?
234
+ * Add ExtractJob#ml_tf_saved_model?
235
+ * Add ExtractJob#ml_xgboost_booster?
236
+ * Add Model#extract and #extract_job
237
+
238
+ ### 1.21.2 / 2020-07-21
239
+
240
+ #### Documentation
241
+
242
+ * Update Data#each samples
243
+
244
+ ### 1.21.1 / 2020-05-28
245
+
246
+ #### Documentation
247
+
248
+ * Fix a few broken links
249
+
250
+ ### 1.21.0 / 2020-03-31
251
+
252
+ #### Features
253
+
254
+ * Add Job#parent_job_id and Job#script_statistics
255
+ * Add parent_job to Project#jobs
256
+ * Add Job#num_child_jobs
257
+ * Add Job#parent_job_id
258
+ * Add Job#script_statistics
259
+
260
+ ### 1.20.0 / 2020-03-11
261
+
262
+ #### Features
263
+
264
+ * Add Range Partitioning
265
+ * Add range partitioning methods to Table and Table::Updater
266
+ * Add range partitioning methods to LoadJob
267
+ * Add range partitioning methods to QueryJob
268
+
269
+ ### 1.19.0 / 2020-02-11
270
+
271
+ #### Features
272
+
273
+ * Add Routine
274
+ * Add Dataset#create_routine
275
+ * Add Argument
276
+ * Update StandardSql classes to expose public initializer
277
+ * Add Data#ddl_target_routine and QueryJob#ddl_target_routine
278
+ * Allow row inserts to skip insert_id generation
279
+ * Streaming inserts using an insert_id are not able to be inserted as fast as inserts without an insert_id
280
+ * Add the ability for users to skip insert_id generation in order to speed up the inserts
281
+ * The default behavior continues to generate insert_id values for each row inserted
282
+ * Add yield documentation for Dataset#insert
283
+
284
+ ### 1.18.1 / 2019-12-18
285
+
286
+ #### Bug Fixes
287
+
288
+ * Fix MonitorMixin usage on Ruby 2.7
289
+ * Ruby 2.7 will error if new_cond is called before super().
290
+ * Make the call to super() be the first call in initialize
291
+
292
+ ### 1.18.0 / 2019-11-06
293
+
294
+ #### Features
295
+
296
+ * Add optional query parameter types
297
+ * Allow query parameters to be nil/NULL when providing an optional
298
+ * Add types argument to the following methods:
299
+ * Project#query
300
+ * Project#query_job
301
+ * Dataset#query
302
+ * Dataset#query_job
303
+ * Add param types helper methods
304
+ * Return the BigQuery field type code, using the same format as the
305
+ * Add Schema::Field#param_type
306
+ * Add Schema#param_types
307
+ * Add Data#param_types
308
+ * Add Table#param_types
309
+ * Add External::CvsSource#param_types
310
+ * Add External::JsonSource#param_types
311
+ * Add support for all_users special role in Dataset access
312
+
313
+ ### 1.17.0 / 2019-10-29
314
+
315
+ This release requires Ruby 2.4 or later.
316
+
317
+ #### Documentation
318
+
319
+ * Clarify which Google Cloud Platform environments support automatic authentication
320
+
321
+ ### 1.16.0 / 2019-10-03
322
+
323
+ #### Features
324
+
325
+ * Add Dataset default_encryption
326
+ * Add Dataset#default_encryption
327
+ * Add Dataset#default_encryption=
328
+
329
+ ### 1.15.0 / 2019-09-30
330
+
331
+ #### Features
332
+
333
+ * Add Model encryption
334
+ * Add Model#encryption
335
+ * Add Model#encryption=
336
+ * Add range support for Google Sheets
337
+ * Add External::SheetsSource#range
338
+ * Add External::SheetsSource#range=
339
+ * Support use_avro_logical_types on extract jobs
340
+ * Add ExtractJob#use_avro_logical_types?
341
+ * Add ExtractJob::Updater#use_avro_logical_types=
342
+
343
+ ### 1.14.1 / 2019-09-04
344
+
345
+ #### Documentation
346
+
347
+ * Add note about streaming insert issues
348
+ * Acknowledge tradeoffs when inserting rows soon after
349
+ table metadata has been changed.
350
+ * Add link to BigQuery Troubleshooting guide.
351
+
352
+ ### 1.14.0 / 2019-08-23
353
+
354
+ #### Features
355
+
356
+ * Support overriding of service endpoint
357
+
358
+ #### Performance Improvements
359
+
360
+ * Use MiniMime to detect content types
361
+
362
+ #### Documentation
363
+
364
+ * Update documentation
365
+
366
+ ### 1.13.0 / 2019-07-31
367
+
368
+ * Add Table#require_partition_filter
369
+ * List jobs using min and max created_at
370
+ * Reduce thread usage at startup
371
+ * Allocate threads in pool as needed, not all up front
372
+ * Update documentation links
373
+
3
374
  ### 1.12.0 / 2019-07-10
4
375
 
5
376
  * Add BigQuery Model API
@@ -34,7 +405,7 @@
34
405
  * Add copy and extract methods to Project
35
406
  * Add Project#extract and Project#extract_job
36
407
  * Add Project#copy and Project#copy_job
37
- * Deprecate dryrun param in Table#copy_job, Table#extract_job and
408
+ * Deprecate dryrun param in Table#copy_job, Table#extract_job and
38
409
  Table#load_job
39
410
  * Fix memoization in Dataset#exists? and Table#exists?
40
411
  * Add force param to Dataset#exists? and Table#exists?