embulk-output-bigquery 0.6.0 → 0.6.5
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +23 -3
- data/Gemfile +1 -1
- data/README.md +16 -11
- data/embulk-output-bigquery.gemspec +15 -3
- data/lib/embulk/output/bigquery.rb +3 -3
- data/lib/embulk/output/bigquery/value_converter_factory.rb +31 -0
- data/test/test_helper.rb +7 -1
- data/test/test_value_converter_factory.rb +86 -0
- metadata +70 -60
- data/example/config_append_direct_schema_update_options.yml +0 -31
- data/example/config_client_options.yml +0 -33
- data/example/config_csv.yml +0 -30
- data/example/config_delete_in_advance.yml +0 -29
- data/example/config_delete_in_advance_field_partitioned_table.yml +0 -33
- data/example/config_delete_in_advance_partitioned_table.yml +0 -33
- data/example/config_expose_errors.yml +0 -30
- data/example/config_gcs.yml +0 -32
- data/example/config_guess_from_embulk_schema.yml +0 -29
- data/example/config_guess_with_column_options.yml +0 -40
- data/example/config_gzip.yml +0 -1
- data/example/config_jsonl.yml +0 -1
- data/example/config_max_threads.yml +0 -34
- data/example/config_min_ouput_tasks.yml +0 -34
- data/example/config_mode_append.yml +0 -30
- data/example/config_mode_append_direct.yml +0 -30
- data/example/config_nested_record.yml +0 -1
- data/example/config_payload_column.yml +0 -20
- data/example/config_payload_column_index.yml +0 -20
- data/example/config_progress_log_interval.yml +0 -31
- data/example/config_replace.yml +0 -30
- data/example/config_replace_backup.yml +0 -32
- data/example/config_replace_backup_field_partitioned_table.yml +0 -34
- data/example/config_replace_backup_partitioned_table.yml +0 -34
- data/example/config_replace_field_partitioned_table.yml +0 -33
- data/example/config_replace_partitioned_table.yml +0 -33
- data/example/config_replace_schema_update_options.yml +0 -33
- data/example/config_skip_file_generation.yml +0 -32
- data/example/config_table_strftime.yml +0 -30
- data/example/config_template_table.yml +0 -21
- data/example/config_uncompressed.yml +0 -1
- data/example/config_with_rehearsal.yml +0 -33
- data/example/example.csv +0 -17
- data/example/example.yml +0 -1
- data/example/example2_1.csv +0 -1
- data/example/example2_2.csv +0 -1
- data/example/example4_1.csv +0 -1
- data/example/example4_2.csv +0 -1
- data/example/example4_3.csv +0 -1
- data/example/example4_4.csv +0 -1
- data/example/json_key.json +0 -12
- data/example/nested_example.jsonl +0 -16
- data/example/schema.json +0 -30
- data/example/schema_expose_errors.json +0 -30
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 1cc922d557454c9b4e50d1bce1ad4625161c6894cbae47a392ed174ad4d2123f
|
4
|
+
data.tar.gz: 285a811634c1a2ddad6e19389ba385910743be4ee598fa03501c6307db40ce4f
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: f510a2cc1ffec9b5ee2965a569b01920b4955a8a9fa784e173e67b47a7cef081dc07f4859d45e9914fe4afcec8f2aaacd4389db891546b818037560e0d9e0907
|
7
|
+
data.tar.gz: 644008c32c635ee128cedde95918bf74d08291d026f2d9edd83d82d406772652b69b4312342f83435ef7dd4af6fa33d434647d4d1d8cea5e086317a6f3815e00
|
data/CHANGELOG.md
CHANGED
@@ -1,3 +1,23 @@
|
|
1
|
+
## 0.6.5 - 2021-06-10
|
2
|
+
* [maintenance] Fix failed tests (thanks to @kyoshidajp)
|
3
|
+
* [maintenance] Lock representable version for avoiding requiring Ruby 2.4 (thanks to @hiroyuki-sato)
|
4
|
+
|
5
|
+
## 0.6.4 - 2019-11-06
|
6
|
+
|
7
|
+
* [enhancement] Add DATETIME type conveter (thanks to @kekekenta)
|
8
|
+
|
9
|
+
## 0.6.3 - 2019-10-28
|
10
|
+
|
11
|
+
* [enhancement] Add DATE type conveter (thanks to @tksfjt1024)
|
12
|
+
|
13
|
+
## 0.6.2 - 2019-10-16
|
14
|
+
|
15
|
+
* [maintenance] Lock signet and google-api-client version (thanks to @hiroyuki-sato)
|
16
|
+
|
17
|
+
## 0.6.1 - 2019-08-28
|
18
|
+
|
19
|
+
* [maintenance] Release a new gem not to include symlinks to make it work on Windows.
|
20
|
+
|
1
21
|
## 0.6.0 - 2019-08-11
|
2
22
|
|
3
23
|
Cleanup `auth_method`:
|
@@ -5,14 +25,14 @@ Cleanup `auth_method`:
|
|
5
25
|
* [enhancement] Support `auth_method: authorized_user` (OAuth)
|
6
26
|
* [incompatibility change] Rename `auth_method: json_key` to `auth_method: service_account` (`json_key` is kept for backward compatibility)
|
7
27
|
* [incompatibility change] Remove deprecated `auth_method: private_key` (p12 key)
|
8
|
-
* [incompatibility change] Change the default `auth_method` to `application_default` from `private_key
|
28
|
+
* [incompatibility change] Change the default `auth_method` to `application_default` from `private_key` because `private_key` was dropped.
|
9
29
|
|
10
30
|
## 0.5.0 - 2019-08-10
|
11
31
|
|
12
32
|
* [incompatibility change] Drop deprecated `time_partitioning`.`require_partition_filter`
|
13
33
|
* [incompatibility change] Drop `prevent_duplicate_insert` which has no use-case now
|
14
|
-
* [incompatibility change]
|
15
|
-
|
34
|
+
* [incompatibility change] Modes `replace`, `replace_backup`, `append`, and `delete_in_advance` require `auto_create_table: true` now because, previously, these modes had created a target table even with `auto_create_table: false` and made users being confused. Note that `auto_create_table: true` is always required even for a partition (a table name with a partition decorator) which may not require creating a table. This is for simplicity of logics and implementations.
|
35
|
+
* [incompatibility change] Change default value of `auto_create_table` to `true` because the above 4 modes, that is, except `append_direct` always require `auto_create_table: true` now.
|
16
36
|
|
17
37
|
## 0.4.14 - 2019-08-10
|
18
38
|
|
data/Gemfile
CHANGED
data/README.md
CHANGED
@@ -37,7 +37,7 @@ OAuth flow for installed applications.
|
|
37
37
|
| location | string | optional | nil | geographic location of dataset. See [Location](#location) |
|
38
38
|
| table | string | required | | table name, or table name with a partition decorator such as `table_name$20160929`|
|
39
39
|
| auto_create_dataset | boolean | optional | false | automatically create dataset |
|
40
|
-
| auto_create_table | boolean | optional | true | `false` is available only for `append_direct` mode. Other modes
|
40
|
+
| auto_create_table | boolean | optional | true | `false` is available only for `append_direct` mode. Other modes require `true`. See [Dynamic Table Creating](#dynamic-table-creating) and [Time Partitioning](#time-partitioning) |
|
41
41
|
| schema_file | string | optional | | /path/to/schema.json |
|
42
42
|
| template_table | string | optional | | template table name. See [Dynamic Table Creating](#dynamic-table-creating) |
|
43
43
|
| job_status_max_polling_time | int | optional | 3600 sec | Max job status polling time |
|
@@ -213,7 +213,7 @@ You can also embed contents of `json_keyfile` at config.yml.
|
|
213
213
|
```yaml
|
214
214
|
out:
|
215
215
|
type: bigquery
|
216
|
-
auth_method:
|
216
|
+
auth_method: authorized_user
|
217
217
|
json_keyfile:
|
218
218
|
content: |
|
219
219
|
{
|
@@ -239,7 +239,12 @@ out:
|
|
239
239
|
|
240
240
|
#### application\_default
|
241
241
|
|
242
|
-
Use Application Default Credentials (ADC).
|
242
|
+
Use Application Default Credentials (ADC). ADC is a strategy to locate Google Cloud Service Account credentials.
|
243
|
+
|
244
|
+
1. ADC checks to see if the environment variable `GOOGLE_APPLICATION_CREDENTIALS` is set. If the variable is set, ADC uses the service account file that the variable points to.
|
245
|
+
2. ADC checks to see if `~/.config/gcloud/application_default_credentials.json` is located. This file is created by running `gcloud auth application-default login`.
|
246
|
+
3. Use the default service account for credentials if the application running on Compute Engine, App Engine, Kubernetes Engine, Cloud Functions or Cloud Run.
|
247
|
+
|
243
248
|
See https://cloud.google.com/docs/authentication/production for details.
|
244
249
|
|
245
250
|
```yaml
|
@@ -256,12 +261,12 @@ Table ids are formatted at runtime
|
|
256
261
|
using the local time of the embulk server.
|
257
262
|
|
258
263
|
For example, with the configuration below,
|
259
|
-
data is inserted into tables `
|
264
|
+
data is inserted into tables `table_20150503`, `table_20150504` and so on.
|
260
265
|
|
261
266
|
```yaml
|
262
267
|
out:
|
263
268
|
type: bigquery
|
264
|
-
table: table_%
|
269
|
+
table: table_%Y%m%d
|
265
270
|
```
|
266
271
|
|
267
272
|
### Dynamic table creating
|
@@ -276,7 +281,7 @@ Please set file path of schema.json.
|
|
276
281
|
out:
|
277
282
|
type: bigquery
|
278
283
|
auto_create_table: true
|
279
|
-
table: table_%
|
284
|
+
table: table_%Y%m%d
|
280
285
|
schema_file: /path/to/schema.json
|
281
286
|
```
|
282
287
|
|
@@ -288,7 +293,7 @@ Plugin will try to read schema from existing table and use it as schema template
|
|
288
293
|
out:
|
289
294
|
type: bigquery
|
290
295
|
auto_create_table: true
|
291
|
-
table: table_%
|
296
|
+
table: table_%Y%m%d
|
292
297
|
template_table: existing_table_name
|
293
298
|
```
|
294
299
|
|
@@ -302,17 +307,17 @@ Column options are used to aid guessing BigQuery schema, or to define conversion
|
|
302
307
|
|
303
308
|
- **column_options**: advanced: an array of options for columns
|
304
309
|
- **name**: column name
|
305
|
-
- **type**: BigQuery type such as `BOOLEAN`, `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP`, and `RECORD`. See belows for supported conversion type.
|
310
|
+
- **type**: BigQuery type such as `BOOLEAN`, `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP`, `DATETIME`, `DATE`, and `RECORD`. See belows for supported conversion type.
|
306
311
|
- boolean: `BOOLEAN`, `STRING` (default: `BOOLEAN`)
|
307
312
|
- long: `BOOLEAN`, `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP` (default: `INTEGER`)
|
308
313
|
- double: `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP` (default: `FLOAT`)
|
309
|
-
- string: `BOOLEAN`, `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP`, `RECORD` (default: `STRING`)
|
310
|
-
- timestamp: `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP` (default: `TIMESTAMP`)
|
314
|
+
- string: `BOOLEAN`, `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP`, `DATETIME`, `DATE`, `RECORD` (default: `STRING`)
|
315
|
+
- timestamp: `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP`, `DATETIME`, `DATE` (default: `TIMESTAMP`)
|
311
316
|
- json: `STRING`, `RECORD` (default: `STRING`)
|
312
317
|
- **mode**: BigQuery mode such as `NULLABLE`, `REQUIRED`, and `REPEATED` (string, default: `NULLABLE`)
|
313
318
|
- **fields**: Describes the nested schema fields if the type property is set to RECORD. Please note that this is **required** for `RECORD` column.
|
314
319
|
- **timestamp_format**: timestamp format to convert into/from `timestamp` (string, default is `default_timestamp_format`)
|
315
|
-
- **timezone**: timezone to convert into/from `timestamp` (string, default is `default_timezone`).
|
320
|
+
- **timezone**: timezone to convert into/from `timestamp`, `date` (string, default is `default_timezone`).
|
316
321
|
- **default_timestamp_format**: default timestamp format for column_options (string, default is "%Y-%m-%d %H:%M:%S.%6N")
|
317
322
|
- **default_timezone**: default timezone for column_options (string, default is "UTC")
|
318
323
|
|
@@ -1,6 +1,6 @@
|
|
1
1
|
Gem::Specification.new do |spec|
|
2
2
|
spec.name = "embulk-output-bigquery"
|
3
|
-
spec.version = "0.6.
|
3
|
+
spec.version = "0.6.5"
|
4
4
|
spec.authors = ["Satoshi Akama", "Naotoshi Seo"]
|
5
5
|
spec.summary = "Google BigQuery output plugin for Embulk"
|
6
6
|
spec.description = "Embulk plugin that insert records to Google BigQuery."
|
@@ -8,12 +8,24 @@ Gem::Specification.new do |spec|
|
|
8
8
|
spec.licenses = ["MIT"]
|
9
9
|
spec.homepage = "https://github.com/embulk/embulk-output-bigquery"
|
10
10
|
|
11
|
-
|
11
|
+
# Exclude example directory which uses symlinks from generating gem.
|
12
|
+
# Symlinks do not work properly on the Windows platform without administrator privilege.
|
13
|
+
spec.files = `git ls-files`.split("\n") + Dir["classpath/*.jar"] - Dir["example/*" ]
|
12
14
|
spec.test_files = spec.files.grep(%r{^(test|spec)/})
|
13
15
|
spec.require_paths = ["lib"]
|
14
16
|
|
15
|
-
|
17
|
+
# TODO
|
18
|
+
# signet 0.12.0 and google-api-client 0.33.0 require >= Ruby 2.4.
|
19
|
+
# Embulk 0.9 use JRuby 9.1.X.Y and It compatible Ruby 2.3.
|
20
|
+
# So, Force install signet < 0.12 and google-api-client < 0.33.0
|
21
|
+
# Also, representable veresion > 3.1.0 requires Ruby version >= 2.4
|
22
|
+
spec.add_dependency 'signet', '~> 0.7', '< 0.12.0'
|
23
|
+
spec.add_dependency 'google-api-client','< 0.33.0'
|
16
24
|
spec.add_dependency 'time_with_zone'
|
25
|
+
spec.add_dependency "representable", ['~> 3.0.0', '< 3.1']
|
26
|
+
# faraday 1.1.0 require >= Ruby 2.4.
|
27
|
+
# googleauth 0.9.0 requires faraday ~> 0.12
|
28
|
+
spec.add_dependency "faraday", '~> 0.12'
|
17
29
|
|
18
30
|
spec.add_development_dependency 'bundler', ['>= 1.10.6']
|
19
31
|
spec.add_development_dependency 'rake', ['>= 10.0']
|
@@ -304,14 +304,14 @@ module Embulk
|
|
304
304
|
bigquery.create_table_if_not_exists(task['table'])
|
305
305
|
when 'replace'
|
306
306
|
bigquery.create_table_if_not_exists(task['temp_table'])
|
307
|
-
bigquery.create_table_if_not_exists(task['table'])
|
307
|
+
bigquery.create_table_if_not_exists(task['table']) # needs for when task['table'] is a partition
|
308
308
|
when 'append'
|
309
309
|
bigquery.create_table_if_not_exists(task['temp_table'])
|
310
|
-
bigquery.create_table_if_not_exists(task['table'])
|
310
|
+
bigquery.create_table_if_not_exists(task['table']) # needs for when task['table'] is a partition
|
311
311
|
when 'replace_backup'
|
312
312
|
bigquery.create_table_if_not_exists(task['temp_table'])
|
313
313
|
bigquery.create_table_if_not_exists(task['table'])
|
314
|
-
bigquery.create_table_if_not_exists(task['table_old'], dataset: task['dataset_old'])
|
314
|
+
bigquery.create_table_if_not_exists(task['table_old'], dataset: task['dataset_old']) # needs for when a partition
|
315
315
|
else # append_direct
|
316
316
|
if task['auto_create_table']
|
317
317
|
bigquery.create_table_if_not_exists(task['table'])
|
@@ -203,6 +203,27 @@ module Embulk
|
|
203
203
|
val # Users must care of BQ timestamp format
|
204
204
|
}
|
205
205
|
end
|
206
|
+
when 'DATE'
|
207
|
+
Proc.new {|val|
|
208
|
+
next nil if val.nil?
|
209
|
+
with_typecast_error(val) do |val|
|
210
|
+
TimeWithZone.set_zone_offset(Time.parse(val), zone_offset).strftime("%Y-%m-%d")
|
211
|
+
end
|
212
|
+
}
|
213
|
+
when 'DATETIME'
|
214
|
+
if @timestamp_format
|
215
|
+
Proc.new {|val|
|
216
|
+
next nil if val.nil?
|
217
|
+
with_typecast_error(val) do |val|
|
218
|
+
Time.strptime(val, @timestamp_format).strftime("%Y-%m-%d %H:%M:%S.%6N")
|
219
|
+
end
|
220
|
+
}
|
221
|
+
else
|
222
|
+
Proc.new {|val|
|
223
|
+
next nil if val.nil?
|
224
|
+
val # Users must care of BQ timestamp format
|
225
|
+
}
|
226
|
+
end
|
206
227
|
when 'RECORD'
|
207
228
|
Proc.new {|val|
|
208
229
|
next nil if val.nil?
|
@@ -240,6 +261,16 @@ module Embulk
|
|
240
261
|
next nil if val.nil?
|
241
262
|
val.strftime("%Y-%m-%d %H:%M:%S.%6N %:z")
|
242
263
|
}
|
264
|
+
when 'DATE'
|
265
|
+
Proc.new {|val|
|
266
|
+
next nil if val.nil?
|
267
|
+
val.localtime(zone_offset).strftime("%Y-%m-%d")
|
268
|
+
}
|
269
|
+
when 'DATETIME'
|
270
|
+
Proc.new {|val|
|
271
|
+
next nil if val.nil?
|
272
|
+
val.localtime(zone_offset).strftime("%Y-%m-%d %H:%M:%S.%6N")
|
273
|
+
}
|
243
274
|
else
|
244
275
|
raise NotSupportedType, "cannot take column type #{type} for timestamp column"
|
245
276
|
end
|
data/test/test_helper.rb
CHANGED
@@ -62,7 +62,9 @@ module Embulk
|
|
62
62
|
Column.new({index: 2, name: 'double', type: :double}),
|
63
63
|
Column.new({index: 3, name: 'string', type: :string}),
|
64
64
|
Column.new({index: 4, name: 'timestamp', type: :timestamp}),
|
65
|
-
Column.new({index: 5, name: '
|
65
|
+
Column.new({index: 5, name: 'date', type: :timestamp}),
|
66
|
+
Column.new({index: 6, name: 'datetime', type: :timestamp}),
|
67
|
+
Column.new({index: 7, name: 'json', type: :json}),
|
66
68
|
])
|
67
69
|
task = {
|
68
70
|
'column_options' => [
|
@@ -71,6 +73,8 @@ module Embulk
|
|
71
73
|
{'name' => 'double', 'type' => 'STRING'},
|
72
74
|
{'name' => 'string', 'type' => 'INTEGER'},
|
73
75
|
{'name' => 'timestamp', 'type' => 'INTEGER'},
|
76
|
+
{'name' => 'date', 'type' => 'DATE'},
|
77
|
+
{'name' => 'datetime', 'type' => 'DATETIME'},
|
74
78
|
{'name' => 'json', 'type' => 'RECORD', 'fields' => [
|
75
79
|
{ 'name' => 'key1', 'type' => 'STRING' },
|
76
80
|
]},
|
@@ -82,6 +86,8 @@ module Embulk
|
|
82
86
|
{name: 'double', type: 'STRING'},
|
83
87
|
{name: 'string', type: 'INTEGER'},
|
84
88
|
{name: 'timestamp', type: 'INTEGER'},
|
89
|
+
{name: 'date', type: 'DATE'},
|
90
|
+
{name: 'datetime', type: 'DATETIME'},
|
85
91
|
{name: 'json', type: 'RECORD', fields: [
|
86
92
|
{name: 'key1', type: 'STRING'},
|
87
93
|
]},
|
@@ -90,6 +90,14 @@ module Embulk
|
|
90
90
|
assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'TIMESTAMP').create_converter }
|
91
91
|
end
|
92
92
|
|
93
|
+
def test_date
|
94
|
+
assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'DATE').create_converter }
|
95
|
+
end
|
96
|
+
|
97
|
+
def test_datetime
|
98
|
+
assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'DATETIME').create_converter }
|
99
|
+
end
|
100
|
+
|
93
101
|
def test_record
|
94
102
|
assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'RECORD').create_converter }
|
95
103
|
end
|
@@ -130,6 +138,14 @@ module Embulk
|
|
130
138
|
assert_equal 1408452095, converter.call(1408452095)
|
131
139
|
end
|
132
140
|
|
141
|
+
def test_date
|
142
|
+
assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'DATE').create_converter }
|
143
|
+
end
|
144
|
+
|
145
|
+
def test_datetime
|
146
|
+
assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'DATETIME').create_converter }
|
147
|
+
end
|
148
|
+
|
133
149
|
def test_record
|
134
150
|
assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'RECORD').create_converter }
|
135
151
|
end
|
@@ -166,6 +182,14 @@ module Embulk
|
|
166
182
|
assert_equal 1408452095.188766, converter.call(1408452095.188766)
|
167
183
|
end
|
168
184
|
|
185
|
+
def test_date
|
186
|
+
assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'DATE').create_converter }
|
187
|
+
end
|
188
|
+
|
189
|
+
def test_datetime
|
190
|
+
assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'DATETIME').create_converter }
|
191
|
+
end
|
192
|
+
|
169
193
|
def test_record
|
170
194
|
assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'RECORD').create_converter }
|
171
195
|
end
|
@@ -216,6 +240,28 @@ module Embulk
|
|
216
240
|
assert_equal "2016-02-26 00:00:00", converter.call("2016-02-26 00:00:00")
|
217
241
|
end
|
218
242
|
|
243
|
+
def test_date
|
244
|
+
converter = ValueConverterFactory.new(SCHEMA_TYPE, 'DATE').create_converter
|
245
|
+
assert_equal nil, converter.call(nil)
|
246
|
+
assert_equal "2016-02-26", converter.call("2016-02-26")
|
247
|
+
assert_equal "2016-02-26", converter.call("2016-02-26 00:00:00")
|
248
|
+
assert_raise { converter.call('foo') }
|
249
|
+
end
|
250
|
+
|
251
|
+
def test_datetime
|
252
|
+
converter = ValueConverterFactory.new(
|
253
|
+
SCHEMA_TYPE, 'DATETIME',
|
254
|
+
timestamp_format: '%Y/%m/%d'
|
255
|
+
).create_converter
|
256
|
+
assert_equal nil, converter.call(nil)
|
257
|
+
assert_equal "2016-02-26 00:00:00.000000", converter.call("2016/02/26")
|
258
|
+
|
259
|
+
# Users must care of BQ datetime format by themselves with no timestamp_format
|
260
|
+
converter = ValueConverterFactory.new(SCHEMA_TYPE, 'DATETIME').create_converter
|
261
|
+
assert_equal nil, converter.call(nil)
|
262
|
+
assert_equal "2016-02-26 00:00:00", converter.call("2016-02-26 00:00:00")
|
263
|
+
end
|
264
|
+
|
219
265
|
def test_record
|
220
266
|
converter = ValueConverterFactory.new(SCHEMA_TYPE, 'RECORD').create_converter
|
221
267
|
assert_equal({'foo'=>'foo'}, converter.call(%Q[{"foo":"foo"}]))
|
@@ -268,6 +314,42 @@ module Embulk
|
|
268
314
|
assert_equal expected, converter.call(Time.at(subject).utc)
|
269
315
|
end
|
270
316
|
|
317
|
+
def test_date
|
318
|
+
converter = ValueConverterFactory.new(SCHEMA_TYPE, 'DATE').create_converter
|
319
|
+
assert_equal nil, converter.call(nil)
|
320
|
+
timestamp = Time.parse("2016-02-26 00:00:00.500000 +00:00")
|
321
|
+
expected = "2016-02-26"
|
322
|
+
assert_equal expected, converter.call(timestamp)
|
323
|
+
|
324
|
+
converter = ValueConverterFactory.new(
|
325
|
+
SCHEMA_TYPE, 'DATE', timezone: 'Asia/Tokyo'
|
326
|
+
).create_converter
|
327
|
+
assert_equal nil, converter.call(nil)
|
328
|
+
timestamp = Time.parse("2016-02-25 15:00:00.500000 +00:00")
|
329
|
+
expected = "2016-02-26"
|
330
|
+
assert_equal expected, converter.call(timestamp)
|
331
|
+
|
332
|
+
assert_raise { converter.call('foo') }
|
333
|
+
end
|
334
|
+
|
335
|
+
def test_datetime
|
336
|
+
converter = ValueConverterFactory.new(SCHEMA_TYPE, 'DATETIME').create_converter
|
337
|
+
assert_equal nil, converter.call(nil)
|
338
|
+
timestamp = Time.parse("2016-02-26 00:00:00.500000 +00:00")
|
339
|
+
expected = "2016-02-26 00:00:00.500000"
|
340
|
+
assert_equal expected, converter.call(timestamp)
|
341
|
+
|
342
|
+
converter = ValueConverterFactory.new(
|
343
|
+
SCHEMA_TYPE, 'DATETIME', timezone: 'Asia/Tokyo'
|
344
|
+
).create_converter
|
345
|
+
assert_equal nil, converter.call(nil)
|
346
|
+
timestamp = Time.parse("2016-02-25 15:00:00.500000 +00:00")
|
347
|
+
expected = "2016-02-26 00:00:00.500000"
|
348
|
+
assert_equal expected, converter.call(timestamp)
|
349
|
+
|
350
|
+
assert_raise { converter.call('foo') }
|
351
|
+
end
|
352
|
+
|
271
353
|
def test_record
|
272
354
|
assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'RECORD').create_converter }
|
273
355
|
end
|
@@ -298,6 +380,10 @@ module Embulk
|
|
298
380
|
assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'TIMESTAMP').create_converter }
|
299
381
|
end
|
300
382
|
|
383
|
+
def test_date
|
384
|
+
assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'DATE').create_converter }
|
385
|
+
end
|
386
|
+
|
301
387
|
def test_record
|
302
388
|
converter = ValueConverterFactory.new(SCHEMA_TYPE, 'RECORD').create_converter
|
303
389
|
assert_equal nil, converter.call(nil)
|
metadata
CHANGED
@@ -1,67 +1,121 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: embulk-output-bigquery
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.6.
|
4
|
+
version: 0.6.5
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Satoshi Akama
|
8
8
|
- Naotoshi Seo
|
9
|
-
autorequire:
|
9
|
+
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date:
|
12
|
+
date: 2021-06-09 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
|
+
name: signet
|
15
16
|
requirement: !ruby/object:Gem::Requirement
|
16
17
|
requirements:
|
17
|
-
- - "
|
18
|
+
- - "~>"
|
18
19
|
- !ruby/object:Gem::Version
|
19
|
-
version: '0'
|
20
|
-
|
20
|
+
version: '0.7'
|
21
|
+
- - "<"
|
22
|
+
- !ruby/object:Gem::Version
|
23
|
+
version: 0.12.0
|
24
|
+
type: :runtime
|
21
25
|
prerelease: false
|
26
|
+
version_requirements: !ruby/object:Gem::Requirement
|
27
|
+
requirements:
|
28
|
+
- - "~>"
|
29
|
+
- !ruby/object:Gem::Version
|
30
|
+
version: '0.7'
|
31
|
+
- - "<"
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: 0.12.0
|
34
|
+
- !ruby/object:Gem::Dependency
|
35
|
+
name: google-api-client
|
36
|
+
requirement: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - "<"
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: 0.33.0
|
22
41
|
type: :runtime
|
42
|
+
prerelease: false
|
23
43
|
version_requirements: !ruby/object:Gem::Requirement
|
24
44
|
requirements:
|
25
|
-
- - "
|
45
|
+
- - "<"
|
26
46
|
- !ruby/object:Gem::Version
|
27
|
-
version:
|
47
|
+
version: 0.33.0
|
28
48
|
- !ruby/object:Gem::Dependency
|
49
|
+
name: time_with_zone
|
29
50
|
requirement: !ruby/object:Gem::Requirement
|
30
51
|
requirements:
|
31
52
|
- - ">="
|
32
53
|
- !ruby/object:Gem::Version
|
33
54
|
version: '0'
|
34
|
-
name: time_with_zone
|
35
|
-
prerelease: false
|
36
55
|
type: :runtime
|
56
|
+
prerelease: false
|
37
57
|
version_requirements: !ruby/object:Gem::Requirement
|
38
58
|
requirements:
|
39
59
|
- - ">="
|
40
60
|
- !ruby/object:Gem::Version
|
41
61
|
version: '0'
|
42
62
|
- !ruby/object:Gem::Dependency
|
63
|
+
name: representable
|
64
|
+
requirement: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - "~>"
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: 3.0.0
|
69
|
+
- - "<"
|
70
|
+
- !ruby/object:Gem::Version
|
71
|
+
version: '3.1'
|
72
|
+
type: :runtime
|
73
|
+
prerelease: false
|
74
|
+
version_requirements: !ruby/object:Gem::Requirement
|
75
|
+
requirements:
|
76
|
+
- - "~>"
|
77
|
+
- !ruby/object:Gem::Version
|
78
|
+
version: 3.0.0
|
79
|
+
- - "<"
|
80
|
+
- !ruby/object:Gem::Version
|
81
|
+
version: '3.1'
|
82
|
+
- !ruby/object:Gem::Dependency
|
83
|
+
name: faraday
|
84
|
+
requirement: !ruby/object:Gem::Requirement
|
85
|
+
requirements:
|
86
|
+
- - "~>"
|
87
|
+
- !ruby/object:Gem::Version
|
88
|
+
version: '0.12'
|
89
|
+
type: :runtime
|
90
|
+
prerelease: false
|
91
|
+
version_requirements: !ruby/object:Gem::Requirement
|
92
|
+
requirements:
|
93
|
+
- - "~>"
|
94
|
+
- !ruby/object:Gem::Version
|
95
|
+
version: '0.12'
|
96
|
+
- !ruby/object:Gem::Dependency
|
97
|
+
name: bundler
|
43
98
|
requirement: !ruby/object:Gem::Requirement
|
44
99
|
requirements:
|
45
100
|
- - ">="
|
46
101
|
- !ruby/object:Gem::Version
|
47
102
|
version: 1.10.6
|
48
|
-
name: bundler
|
49
|
-
prerelease: false
|
50
103
|
type: :development
|
104
|
+
prerelease: false
|
51
105
|
version_requirements: !ruby/object:Gem::Requirement
|
52
106
|
requirements:
|
53
107
|
- - ">="
|
54
108
|
- !ruby/object:Gem::Version
|
55
109
|
version: 1.10.6
|
56
110
|
- !ruby/object:Gem::Dependency
|
111
|
+
name: rake
|
57
112
|
requirement: !ruby/object:Gem::Requirement
|
58
113
|
requirements:
|
59
114
|
- - ">="
|
60
115
|
- !ruby/object:Gem::Version
|
61
116
|
version: '10.0'
|
62
|
-
name: rake
|
63
|
-
prerelease: false
|
64
117
|
type: :development
|
118
|
+
prerelease: false
|
65
119
|
version_requirements: !ruby/object:Gem::Requirement
|
66
120
|
requirements:
|
67
121
|
- - ">="
|
@@ -83,50 +137,6 @@ files:
|
|
83
137
|
- README.md
|
84
138
|
- Rakefile
|
85
139
|
- embulk-output-bigquery.gemspec
|
86
|
-
- example/config_append_direct_schema_update_options.yml
|
87
|
-
- example/config_client_options.yml
|
88
|
-
- example/config_csv.yml
|
89
|
-
- example/config_delete_in_advance.yml
|
90
|
-
- example/config_delete_in_advance_field_partitioned_table.yml
|
91
|
-
- example/config_delete_in_advance_partitioned_table.yml
|
92
|
-
- example/config_expose_errors.yml
|
93
|
-
- example/config_gcs.yml
|
94
|
-
- example/config_guess_from_embulk_schema.yml
|
95
|
-
- example/config_guess_with_column_options.yml
|
96
|
-
- example/config_gzip.yml
|
97
|
-
- example/config_jsonl.yml
|
98
|
-
- example/config_max_threads.yml
|
99
|
-
- example/config_min_ouput_tasks.yml
|
100
|
-
- example/config_mode_append.yml
|
101
|
-
- example/config_mode_append_direct.yml
|
102
|
-
- example/config_nested_record.yml
|
103
|
-
- example/config_payload_column.yml
|
104
|
-
- example/config_payload_column_index.yml
|
105
|
-
- example/config_progress_log_interval.yml
|
106
|
-
- example/config_replace.yml
|
107
|
-
- example/config_replace_backup.yml
|
108
|
-
- example/config_replace_backup_field_partitioned_table.yml
|
109
|
-
- example/config_replace_backup_partitioned_table.yml
|
110
|
-
- example/config_replace_field_partitioned_table.yml
|
111
|
-
- example/config_replace_partitioned_table.yml
|
112
|
-
- example/config_replace_schema_update_options.yml
|
113
|
-
- example/config_skip_file_generation.yml
|
114
|
-
- example/config_table_strftime.yml
|
115
|
-
- example/config_template_table.yml
|
116
|
-
- example/config_uncompressed.yml
|
117
|
-
- example/config_with_rehearsal.yml
|
118
|
-
- example/example.csv
|
119
|
-
- example/example.yml
|
120
|
-
- example/example2_1.csv
|
121
|
-
- example/example2_2.csv
|
122
|
-
- example/example4_1.csv
|
123
|
-
- example/example4_2.csv
|
124
|
-
- example/example4_3.csv
|
125
|
-
- example/example4_4.csv
|
126
|
-
- example/json_key.json
|
127
|
-
- example/nested_example.jsonl
|
128
|
-
- example/schema.json
|
129
|
-
- example/schema_expose_errors.json
|
130
140
|
- lib/embulk/output/bigquery.rb
|
131
141
|
- lib/embulk/output/bigquery/auth.rb
|
132
142
|
- lib/embulk/output/bigquery/bigquery_client.rb
|
@@ -147,7 +157,7 @@ homepage: https://github.com/embulk/embulk-output-bigquery
|
|
147
157
|
licenses:
|
148
158
|
- MIT
|
149
159
|
metadata: {}
|
150
|
-
post_install_message:
|
160
|
+
post_install_message:
|
151
161
|
rdoc_options: []
|
152
162
|
require_paths:
|
153
163
|
- lib
|
@@ -163,7 +173,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
163
173
|
version: '0'
|
164
174
|
requirements: []
|
165
175
|
rubygems_version: 3.0.3
|
166
|
-
signing_key:
|
176
|
+
signing_key:
|
167
177
|
specification_version: 4
|
168
178
|
summary: Google BigQuery output plugin for Embulk
|
169
179
|
test_files:
|