embulk-output-bigquery 0.6.0 → 0.6.5

Sign up to get free protection for your applications and to get access to all the features.
Files changed (54) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +23 -3
  3. data/Gemfile +1 -1
  4. data/README.md +16 -11
  5. data/embulk-output-bigquery.gemspec +15 -3
  6. data/lib/embulk/output/bigquery.rb +3 -3
  7. data/lib/embulk/output/bigquery/value_converter_factory.rb +31 -0
  8. data/test/test_helper.rb +7 -1
  9. data/test/test_value_converter_factory.rb +86 -0
  10. metadata +70 -60
  11. data/example/config_append_direct_schema_update_options.yml +0 -31
  12. data/example/config_client_options.yml +0 -33
  13. data/example/config_csv.yml +0 -30
  14. data/example/config_delete_in_advance.yml +0 -29
  15. data/example/config_delete_in_advance_field_partitioned_table.yml +0 -33
  16. data/example/config_delete_in_advance_partitioned_table.yml +0 -33
  17. data/example/config_expose_errors.yml +0 -30
  18. data/example/config_gcs.yml +0 -32
  19. data/example/config_guess_from_embulk_schema.yml +0 -29
  20. data/example/config_guess_with_column_options.yml +0 -40
  21. data/example/config_gzip.yml +0 -1
  22. data/example/config_jsonl.yml +0 -1
  23. data/example/config_max_threads.yml +0 -34
  24. data/example/config_min_ouput_tasks.yml +0 -34
  25. data/example/config_mode_append.yml +0 -30
  26. data/example/config_mode_append_direct.yml +0 -30
  27. data/example/config_nested_record.yml +0 -1
  28. data/example/config_payload_column.yml +0 -20
  29. data/example/config_payload_column_index.yml +0 -20
  30. data/example/config_progress_log_interval.yml +0 -31
  31. data/example/config_replace.yml +0 -30
  32. data/example/config_replace_backup.yml +0 -32
  33. data/example/config_replace_backup_field_partitioned_table.yml +0 -34
  34. data/example/config_replace_backup_partitioned_table.yml +0 -34
  35. data/example/config_replace_field_partitioned_table.yml +0 -33
  36. data/example/config_replace_partitioned_table.yml +0 -33
  37. data/example/config_replace_schema_update_options.yml +0 -33
  38. data/example/config_skip_file_generation.yml +0 -32
  39. data/example/config_table_strftime.yml +0 -30
  40. data/example/config_template_table.yml +0 -21
  41. data/example/config_uncompressed.yml +0 -1
  42. data/example/config_with_rehearsal.yml +0 -33
  43. data/example/example.csv +0 -17
  44. data/example/example.yml +0 -1
  45. data/example/example2_1.csv +0 -1
  46. data/example/example2_2.csv +0 -1
  47. data/example/example4_1.csv +0 -1
  48. data/example/example4_2.csv +0 -1
  49. data/example/example4_3.csv +0 -1
  50. data/example/example4_4.csv +0 -1
  51. data/example/json_key.json +0 -12
  52. data/example/nested_example.jsonl +0 -16
  53. data/example/schema.json +0 -30
  54. data/example/schema_expose_errors.json +0 -30
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 3c0942035a81c9180260f8329ccaa5ba99de2185ea5f9ec5f1b3ffe87d5e8a73
4
- data.tar.gz: c543a1b9f1278cf5d543a96bd3b8c465b2727b03df67d1e1726bef40135a1d42
3
+ metadata.gz: 1cc922d557454c9b4e50d1bce1ad4625161c6894cbae47a392ed174ad4d2123f
4
+ data.tar.gz: 285a811634c1a2ddad6e19389ba385910743be4ee598fa03501c6307db40ce4f
5
5
  SHA512:
6
- metadata.gz: 23559e485346f2f8d65fa76aef2284c8b8c682257ee317b2088b30515e6e2a2936cc3c5b8ab5c3020ee9f9790e735bc48c41bb8e5d30fc777174d681796128c1
7
- data.tar.gz: 336988c0afb153c0b9b7532bf9d85523bb9e9641eca7a79b6ab491be5567e2be9204c47417b28a8d42bbae2907cb6892b1ce5abb98261564c696b15478deb3ad
6
+ metadata.gz: f510a2cc1ffec9b5ee2965a569b01920b4955a8a9fa784e173e67b47a7cef081dc07f4859d45e9914fe4afcec8f2aaacd4389db891546b818037560e0d9e0907
7
+ data.tar.gz: 644008c32c635ee128cedde95918bf74d08291d026f2d9edd83d82d406772652b69b4312342f83435ef7dd4af6fa33d434647d4d1d8cea5e086317a6f3815e00
data/CHANGELOG.md CHANGED
@@ -1,3 +1,23 @@
1
+ ## 0.6.5 - 2021-06-10
2
+ * [maintenance] Fix failed tests (thanks to @kyoshidajp)
3
+ * [maintenance] Lock representable version for avoiding requiring Ruby 2.4 (thanks to @hiroyuki-sato)
4
+
5
+ ## 0.6.4 - 2019-11-06
6
+
7
+ * [enhancement] Add DATETIME type conveter (thanks to @kekekenta)
8
+
9
+ ## 0.6.3 - 2019-10-28
10
+
11
+ * [enhancement] Add DATE type conveter (thanks to @tksfjt1024)
12
+
13
+ ## 0.6.2 - 2019-10-16
14
+
15
+ * [maintenance] Lock signet and google-api-client version (thanks to @hiroyuki-sato)
16
+
17
+ ## 0.6.1 - 2019-08-28
18
+
19
+ * [maintenance] Release a new gem not to include symlinks to make it work on Windows.
20
+
1
21
  ## 0.6.0 - 2019-08-11
2
22
 
3
23
  Cleanup `auth_method`:
@@ -5,14 +25,14 @@ Cleanup `auth_method`:
5
25
  * [enhancement] Support `auth_method: authorized_user` (OAuth)
6
26
  * [incompatibility change] Rename `auth_method: json_key` to `auth_method: service_account` (`json_key` is kept for backward compatibility)
7
27
  * [incompatibility change] Remove deprecated `auth_method: private_key` (p12 key)
8
- * [incompatibility change] Change the default `auth_method` to `application_default` from `private_key`.
28
+ * [incompatibility change] Change the default `auth_method` to `application_default` from `private_key` because `private_key` was dropped.
9
29
 
10
30
  ## 0.5.0 - 2019-08-10
11
31
 
12
32
  * [incompatibility change] Drop deprecated `time_partitioning`.`require_partition_filter`
13
33
  * [incompatibility change] Drop `prevent_duplicate_insert` which has no use-case now
14
- * [incompatibility change] Change default value of `auto_create_table` to `true` from `false`
15
- * Modes `replace`, `replace_backup`, `append`, `delete_in_advance`, that is, except `append_direct` requires `auto_create_table: true`.
34
+ * [incompatibility change] Modes `replace`, `replace_backup`, `append`, and `delete_in_advance` require `auto_create_table: true` now because, previously, these modes had created a target table even with `auto_create_table: false` and made users being confused. Note that `auto_create_table: true` is always required even for a partition (a table name with a partition decorator) which may not require creating a table. This is for simplicity of logics and implementations.
35
+ * [incompatibility change] Change default value of `auto_create_table` to `true` because the above 4 modes, that is, except `append_direct` always require `auto_create_table: true` now.
16
36
 
17
37
  ## 0.4.14 - 2019-08-10
18
38
 
data/Gemfile CHANGED
@@ -1,7 +1,7 @@
1
1
  source 'https://rubygems.org/'
2
2
 
3
3
  gemspec
4
- gem 'embulk'
4
+ gem 'embulk', '< 0.10'
5
5
  gem 'liquid', '= 4.0.0' # the version included in embulk.jar
6
6
  gem 'embulk-parser-none'
7
7
  gem 'embulk-parser-jsonl'
data/README.md CHANGED
@@ -37,7 +37,7 @@ OAuth flow for installed applications.
37
37
  | location | string | optional | nil | geographic location of dataset. See [Location](#location) |
38
38
  | table | string | required | | table name, or table name with a partition decorator such as `table_name$20160929`|
39
39
  | auto_create_dataset | boolean | optional | false | automatically create dataset |
40
- | auto_create_table | boolean | optional | true | `false` is available only for `append_direct` mode. Other modes requires `true`. See [Dynamic Table Creating](#dynamic-table-creating) and [Time Partitioning](#time-partitioning) |
40
+ | auto_create_table | boolean | optional | true | `false` is available only for `append_direct` mode. Other modes require `true`. See [Dynamic Table Creating](#dynamic-table-creating) and [Time Partitioning](#time-partitioning) |
41
41
  | schema_file | string | optional | | /path/to/schema.json |
42
42
  | template_table | string | optional | | template table name. See [Dynamic Table Creating](#dynamic-table-creating) |
43
43
  | job_status_max_polling_time | int | optional | 3600 sec | Max job status polling time |
@@ -213,7 +213,7 @@ You can also embed contents of `json_keyfile` at config.yml.
213
213
  ```yaml
214
214
  out:
215
215
  type: bigquery
216
- auth_method: service_account
216
+ auth_method: authorized_user
217
217
  json_keyfile:
218
218
  content: |
219
219
  {
@@ -239,7 +239,12 @@ out:
239
239
 
240
240
  #### application\_default
241
241
 
242
- Use Application Default Credentials (ADC).
242
+ Use Application Default Credentials (ADC). ADC is a strategy to locate Google Cloud Service Account credentials.
243
+
244
+ 1. ADC checks to see if the environment variable `GOOGLE_APPLICATION_CREDENTIALS` is set. If the variable is set, ADC uses the service account file that the variable points to.
245
+ 2. ADC checks to see if `~/.config/gcloud/application_default_credentials.json` is located. This file is created by running `gcloud auth application-default login`.
246
+ 3. Use the default service account for credentials if the application running on Compute Engine, App Engine, Kubernetes Engine, Cloud Functions or Cloud Run.
247
+
243
248
  See https://cloud.google.com/docs/authentication/production for details.
244
249
 
245
250
  ```yaml
@@ -256,12 +261,12 @@ Table ids are formatted at runtime
256
261
  using the local time of the embulk server.
257
262
 
258
263
  For example, with the configuration below,
259
- data is inserted into tables `table_2015_04`, `table_2015_05` and so on.
264
+ data is inserted into tables `table_20150503`, `table_20150504` and so on.
260
265
 
261
266
  ```yaml
262
267
  out:
263
268
  type: bigquery
264
- table: table_%Y_%m
269
+ table: table_%Y%m%d
265
270
  ```
266
271
 
267
272
  ### Dynamic table creating
@@ -276,7 +281,7 @@ Please set file path of schema.json.
276
281
  out:
277
282
  type: bigquery
278
283
  auto_create_table: true
279
- table: table_%Y_%m
284
+ table: table_%Y%m%d
280
285
  schema_file: /path/to/schema.json
281
286
  ```
282
287
 
@@ -288,7 +293,7 @@ Plugin will try to read schema from existing table and use it as schema template
288
293
  out:
289
294
  type: bigquery
290
295
  auto_create_table: true
291
- table: table_%Y_%m
296
+ table: table_%Y%m%d
292
297
  template_table: existing_table_name
293
298
  ```
294
299
 
@@ -302,17 +307,17 @@ Column options are used to aid guessing BigQuery schema, or to define conversion
302
307
 
303
308
  - **column_options**: advanced: an array of options for columns
304
309
  - **name**: column name
305
- - **type**: BigQuery type such as `BOOLEAN`, `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP`, and `RECORD`. See belows for supported conversion type.
310
+ - **type**: BigQuery type such as `BOOLEAN`, `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP`, `DATETIME`, `DATE`, and `RECORD`. See belows for supported conversion type.
306
311
  - boolean: `BOOLEAN`, `STRING` (default: `BOOLEAN`)
307
312
  - long: `BOOLEAN`, `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP` (default: `INTEGER`)
308
313
  - double: `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP` (default: `FLOAT`)
309
- - string: `BOOLEAN`, `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP`, `RECORD` (default: `STRING`)
310
- - timestamp: `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP` (default: `TIMESTAMP`)
314
+ - string: `BOOLEAN`, `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP`, `DATETIME`, `DATE`, `RECORD` (default: `STRING`)
315
+ - timestamp: `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP`, `DATETIME`, `DATE` (default: `TIMESTAMP`)
311
316
  - json: `STRING`, `RECORD` (default: `STRING`)
312
317
  - **mode**: BigQuery mode such as `NULLABLE`, `REQUIRED`, and `REPEATED` (string, default: `NULLABLE`)
313
318
  - **fields**: Describes the nested schema fields if the type property is set to RECORD. Please note that this is **required** for `RECORD` column.
314
319
  - **timestamp_format**: timestamp format to convert into/from `timestamp` (string, default is `default_timestamp_format`)
315
- - **timezone**: timezone to convert into/from `timestamp` (string, default is `default_timezone`).
320
+ - **timezone**: timezone to convert into/from `timestamp`, `date` (string, default is `default_timezone`).
316
321
  - **default_timestamp_format**: default timestamp format for column_options (string, default is "%Y-%m-%d %H:%M:%S.%6N")
317
322
  - **default_timezone**: default timezone for column_options (string, default is "UTC")
318
323
 
@@ -1,6 +1,6 @@
1
1
  Gem::Specification.new do |spec|
2
2
  spec.name = "embulk-output-bigquery"
3
- spec.version = "0.6.0"
3
+ spec.version = "0.6.5"
4
4
  spec.authors = ["Satoshi Akama", "Naotoshi Seo"]
5
5
  spec.summary = "Google BigQuery output plugin for Embulk"
6
6
  spec.description = "Embulk plugin that insert records to Google BigQuery."
@@ -8,12 +8,24 @@ Gem::Specification.new do |spec|
8
8
  spec.licenses = ["MIT"]
9
9
  spec.homepage = "https://github.com/embulk/embulk-output-bigquery"
10
10
 
11
- spec.files = `git ls-files`.split("\n") + Dir["classpath/*.jar"]
11
+ # Exclude example directory which uses symlinks from generating gem.
12
+ # Symlinks do not work properly on the Windows platform without administrator privilege.
13
+ spec.files = `git ls-files`.split("\n") + Dir["classpath/*.jar"] - Dir["example/*" ]
12
14
  spec.test_files = spec.files.grep(%r{^(test|spec)/})
13
15
  spec.require_paths = ["lib"]
14
16
 
15
- spec.add_dependency 'google-api-client'
17
+ # TODO
18
+ # signet 0.12.0 and google-api-client 0.33.0 require >= Ruby 2.4.
19
+ # Embulk 0.9 use JRuby 9.1.X.Y and It compatible Ruby 2.3.
20
+ # So, Force install signet < 0.12 and google-api-client < 0.33.0
21
+ # Also, representable veresion > 3.1.0 requires Ruby version >= 2.4
22
+ spec.add_dependency 'signet', '~> 0.7', '< 0.12.0'
23
+ spec.add_dependency 'google-api-client','< 0.33.0'
16
24
  spec.add_dependency 'time_with_zone'
25
+ spec.add_dependency "representable", ['~> 3.0.0', '< 3.1']
26
+ # faraday 1.1.0 require >= Ruby 2.4.
27
+ # googleauth 0.9.0 requires faraday ~> 0.12
28
+ spec.add_dependency "faraday", '~> 0.12'
17
29
 
18
30
  spec.add_development_dependency 'bundler', ['>= 1.10.6']
19
31
  spec.add_development_dependency 'rake', ['>= 10.0']
@@ -304,14 +304,14 @@ module Embulk
304
304
  bigquery.create_table_if_not_exists(task['table'])
305
305
  when 'replace'
306
306
  bigquery.create_table_if_not_exists(task['temp_table'])
307
- bigquery.create_table_if_not_exists(task['table'])
307
+ bigquery.create_table_if_not_exists(task['table']) # needs for when task['table'] is a partition
308
308
  when 'append'
309
309
  bigquery.create_table_if_not_exists(task['temp_table'])
310
- bigquery.create_table_if_not_exists(task['table'])
310
+ bigquery.create_table_if_not_exists(task['table']) # needs for when task['table'] is a partition
311
311
  when 'replace_backup'
312
312
  bigquery.create_table_if_not_exists(task['temp_table'])
313
313
  bigquery.create_table_if_not_exists(task['table'])
314
- bigquery.create_table_if_not_exists(task['table_old'], dataset: task['dataset_old'])
314
+ bigquery.create_table_if_not_exists(task['table_old'], dataset: task['dataset_old']) # needs for when a partition
315
315
  else # append_direct
316
316
  if task['auto_create_table']
317
317
  bigquery.create_table_if_not_exists(task['table'])
@@ -203,6 +203,27 @@ module Embulk
203
203
  val # Users must care of BQ timestamp format
204
204
  }
205
205
  end
206
+ when 'DATE'
207
+ Proc.new {|val|
208
+ next nil if val.nil?
209
+ with_typecast_error(val) do |val|
210
+ TimeWithZone.set_zone_offset(Time.parse(val), zone_offset).strftime("%Y-%m-%d")
211
+ end
212
+ }
213
+ when 'DATETIME'
214
+ if @timestamp_format
215
+ Proc.new {|val|
216
+ next nil if val.nil?
217
+ with_typecast_error(val) do |val|
218
+ Time.strptime(val, @timestamp_format).strftime("%Y-%m-%d %H:%M:%S.%6N")
219
+ end
220
+ }
221
+ else
222
+ Proc.new {|val|
223
+ next nil if val.nil?
224
+ val # Users must care of BQ timestamp format
225
+ }
226
+ end
206
227
  when 'RECORD'
207
228
  Proc.new {|val|
208
229
  next nil if val.nil?
@@ -240,6 +261,16 @@ module Embulk
240
261
  next nil if val.nil?
241
262
  val.strftime("%Y-%m-%d %H:%M:%S.%6N %:z")
242
263
  }
264
+ when 'DATE'
265
+ Proc.new {|val|
266
+ next nil if val.nil?
267
+ val.localtime(zone_offset).strftime("%Y-%m-%d")
268
+ }
269
+ when 'DATETIME'
270
+ Proc.new {|val|
271
+ next nil if val.nil?
272
+ val.localtime(zone_offset).strftime("%Y-%m-%d %H:%M:%S.%6N")
273
+ }
243
274
  else
244
275
  raise NotSupportedType, "cannot take column type #{type} for timestamp column"
245
276
  end
data/test/test_helper.rb CHANGED
@@ -62,7 +62,9 @@ module Embulk
62
62
  Column.new({index: 2, name: 'double', type: :double}),
63
63
  Column.new({index: 3, name: 'string', type: :string}),
64
64
  Column.new({index: 4, name: 'timestamp', type: :timestamp}),
65
- Column.new({index: 5, name: 'json', type: :json}),
65
+ Column.new({index: 5, name: 'date', type: :timestamp}),
66
+ Column.new({index: 6, name: 'datetime', type: :timestamp}),
67
+ Column.new({index: 7, name: 'json', type: :json}),
66
68
  ])
67
69
  task = {
68
70
  'column_options' => [
@@ -71,6 +73,8 @@ module Embulk
71
73
  {'name' => 'double', 'type' => 'STRING'},
72
74
  {'name' => 'string', 'type' => 'INTEGER'},
73
75
  {'name' => 'timestamp', 'type' => 'INTEGER'},
76
+ {'name' => 'date', 'type' => 'DATE'},
77
+ {'name' => 'datetime', 'type' => 'DATETIME'},
74
78
  {'name' => 'json', 'type' => 'RECORD', 'fields' => [
75
79
  { 'name' => 'key1', 'type' => 'STRING' },
76
80
  ]},
@@ -82,6 +86,8 @@ module Embulk
82
86
  {name: 'double', type: 'STRING'},
83
87
  {name: 'string', type: 'INTEGER'},
84
88
  {name: 'timestamp', type: 'INTEGER'},
89
+ {name: 'date', type: 'DATE'},
90
+ {name: 'datetime', type: 'DATETIME'},
85
91
  {name: 'json', type: 'RECORD', fields: [
86
92
  {name: 'key1', type: 'STRING'},
87
93
  ]},
@@ -90,6 +90,14 @@ module Embulk
90
90
  assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'TIMESTAMP').create_converter }
91
91
  end
92
92
 
93
+ def test_date
94
+ assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'DATE').create_converter }
95
+ end
96
+
97
+ def test_datetime
98
+ assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'DATETIME').create_converter }
99
+ end
100
+
93
101
  def test_record
94
102
  assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'RECORD').create_converter }
95
103
  end
@@ -130,6 +138,14 @@ module Embulk
130
138
  assert_equal 1408452095, converter.call(1408452095)
131
139
  end
132
140
 
141
+ def test_date
142
+ assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'DATE').create_converter }
143
+ end
144
+
145
+ def test_datetime
146
+ assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'DATETIME').create_converter }
147
+ end
148
+
133
149
  def test_record
134
150
  assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'RECORD').create_converter }
135
151
  end
@@ -166,6 +182,14 @@ module Embulk
166
182
  assert_equal 1408452095.188766, converter.call(1408452095.188766)
167
183
  end
168
184
 
185
+ def test_date
186
+ assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'DATE').create_converter }
187
+ end
188
+
189
+ def test_datetime
190
+ assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'DATETIME').create_converter }
191
+ end
192
+
169
193
  def test_record
170
194
  assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'RECORD').create_converter }
171
195
  end
@@ -216,6 +240,28 @@ module Embulk
216
240
  assert_equal "2016-02-26 00:00:00", converter.call("2016-02-26 00:00:00")
217
241
  end
218
242
 
243
+ def test_date
244
+ converter = ValueConverterFactory.new(SCHEMA_TYPE, 'DATE').create_converter
245
+ assert_equal nil, converter.call(nil)
246
+ assert_equal "2016-02-26", converter.call("2016-02-26")
247
+ assert_equal "2016-02-26", converter.call("2016-02-26 00:00:00")
248
+ assert_raise { converter.call('foo') }
249
+ end
250
+
251
+ def test_datetime
252
+ converter = ValueConverterFactory.new(
253
+ SCHEMA_TYPE, 'DATETIME',
254
+ timestamp_format: '%Y/%m/%d'
255
+ ).create_converter
256
+ assert_equal nil, converter.call(nil)
257
+ assert_equal "2016-02-26 00:00:00.000000", converter.call("2016/02/26")
258
+
259
+ # Users must care of BQ datetime format by themselves with no timestamp_format
260
+ converter = ValueConverterFactory.new(SCHEMA_TYPE, 'DATETIME').create_converter
261
+ assert_equal nil, converter.call(nil)
262
+ assert_equal "2016-02-26 00:00:00", converter.call("2016-02-26 00:00:00")
263
+ end
264
+
219
265
  def test_record
220
266
  converter = ValueConverterFactory.new(SCHEMA_TYPE, 'RECORD').create_converter
221
267
  assert_equal({'foo'=>'foo'}, converter.call(%Q[{"foo":"foo"}]))
@@ -268,6 +314,42 @@ module Embulk
268
314
  assert_equal expected, converter.call(Time.at(subject).utc)
269
315
  end
270
316
 
317
+ def test_date
318
+ converter = ValueConverterFactory.new(SCHEMA_TYPE, 'DATE').create_converter
319
+ assert_equal nil, converter.call(nil)
320
+ timestamp = Time.parse("2016-02-26 00:00:00.500000 +00:00")
321
+ expected = "2016-02-26"
322
+ assert_equal expected, converter.call(timestamp)
323
+
324
+ converter = ValueConverterFactory.new(
325
+ SCHEMA_TYPE, 'DATE', timezone: 'Asia/Tokyo'
326
+ ).create_converter
327
+ assert_equal nil, converter.call(nil)
328
+ timestamp = Time.parse("2016-02-25 15:00:00.500000 +00:00")
329
+ expected = "2016-02-26"
330
+ assert_equal expected, converter.call(timestamp)
331
+
332
+ assert_raise { converter.call('foo') }
333
+ end
334
+
335
+ def test_datetime
336
+ converter = ValueConverterFactory.new(SCHEMA_TYPE, 'DATETIME').create_converter
337
+ assert_equal nil, converter.call(nil)
338
+ timestamp = Time.parse("2016-02-26 00:00:00.500000 +00:00")
339
+ expected = "2016-02-26 00:00:00.500000"
340
+ assert_equal expected, converter.call(timestamp)
341
+
342
+ converter = ValueConverterFactory.new(
343
+ SCHEMA_TYPE, 'DATETIME', timezone: 'Asia/Tokyo'
344
+ ).create_converter
345
+ assert_equal nil, converter.call(nil)
346
+ timestamp = Time.parse("2016-02-25 15:00:00.500000 +00:00")
347
+ expected = "2016-02-26 00:00:00.500000"
348
+ assert_equal expected, converter.call(timestamp)
349
+
350
+ assert_raise { converter.call('foo') }
351
+ end
352
+
271
353
  def test_record
272
354
  assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'RECORD').create_converter }
273
355
  end
@@ -298,6 +380,10 @@ module Embulk
298
380
  assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'TIMESTAMP').create_converter }
299
381
  end
300
382
 
383
+ def test_date
384
+ assert_raise { ValueConverterFactory.new(SCHEMA_TYPE, 'DATE').create_converter }
385
+ end
386
+
301
387
  def test_record
302
388
  converter = ValueConverterFactory.new(SCHEMA_TYPE, 'RECORD').create_converter
303
389
  assert_equal nil, converter.call(nil)
metadata CHANGED
@@ -1,67 +1,121 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: embulk-output-bigquery
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.6.0
4
+ version: 0.6.5
5
5
  platform: ruby
6
6
  authors:
7
7
  - Satoshi Akama
8
8
  - Naotoshi Seo
9
- autorequire:
9
+ autorequire:
10
10
  bindir: bin
11
11
  cert_chain: []
12
- date: 2019-08-10 00:00:00.000000000 Z
12
+ date: 2021-06-09 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
+ name: signet
15
16
  requirement: !ruby/object:Gem::Requirement
16
17
  requirements:
17
- - - ">="
18
+ - - "~>"
18
19
  - !ruby/object:Gem::Version
19
- version: '0'
20
- name: google-api-client
20
+ version: '0.7'
21
+ - - "<"
22
+ - !ruby/object:Gem::Version
23
+ version: 0.12.0
24
+ type: :runtime
21
25
  prerelease: false
26
+ version_requirements: !ruby/object:Gem::Requirement
27
+ requirements:
28
+ - - "~>"
29
+ - !ruby/object:Gem::Version
30
+ version: '0.7'
31
+ - - "<"
32
+ - !ruby/object:Gem::Version
33
+ version: 0.12.0
34
+ - !ruby/object:Gem::Dependency
35
+ name: google-api-client
36
+ requirement: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - "<"
39
+ - !ruby/object:Gem::Version
40
+ version: 0.33.0
22
41
  type: :runtime
42
+ prerelease: false
23
43
  version_requirements: !ruby/object:Gem::Requirement
24
44
  requirements:
25
- - - ">="
45
+ - - "<"
26
46
  - !ruby/object:Gem::Version
27
- version: '0'
47
+ version: 0.33.0
28
48
  - !ruby/object:Gem::Dependency
49
+ name: time_with_zone
29
50
  requirement: !ruby/object:Gem::Requirement
30
51
  requirements:
31
52
  - - ">="
32
53
  - !ruby/object:Gem::Version
33
54
  version: '0'
34
- name: time_with_zone
35
- prerelease: false
36
55
  type: :runtime
56
+ prerelease: false
37
57
  version_requirements: !ruby/object:Gem::Requirement
38
58
  requirements:
39
59
  - - ">="
40
60
  - !ruby/object:Gem::Version
41
61
  version: '0'
42
62
  - !ruby/object:Gem::Dependency
63
+ name: representable
64
+ requirement: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - "~>"
67
+ - !ruby/object:Gem::Version
68
+ version: 3.0.0
69
+ - - "<"
70
+ - !ruby/object:Gem::Version
71
+ version: '3.1'
72
+ type: :runtime
73
+ prerelease: false
74
+ version_requirements: !ruby/object:Gem::Requirement
75
+ requirements:
76
+ - - "~>"
77
+ - !ruby/object:Gem::Version
78
+ version: 3.0.0
79
+ - - "<"
80
+ - !ruby/object:Gem::Version
81
+ version: '3.1'
82
+ - !ruby/object:Gem::Dependency
83
+ name: faraday
84
+ requirement: !ruby/object:Gem::Requirement
85
+ requirements:
86
+ - - "~>"
87
+ - !ruby/object:Gem::Version
88
+ version: '0.12'
89
+ type: :runtime
90
+ prerelease: false
91
+ version_requirements: !ruby/object:Gem::Requirement
92
+ requirements:
93
+ - - "~>"
94
+ - !ruby/object:Gem::Version
95
+ version: '0.12'
96
+ - !ruby/object:Gem::Dependency
97
+ name: bundler
43
98
  requirement: !ruby/object:Gem::Requirement
44
99
  requirements:
45
100
  - - ">="
46
101
  - !ruby/object:Gem::Version
47
102
  version: 1.10.6
48
- name: bundler
49
- prerelease: false
50
103
  type: :development
104
+ prerelease: false
51
105
  version_requirements: !ruby/object:Gem::Requirement
52
106
  requirements:
53
107
  - - ">="
54
108
  - !ruby/object:Gem::Version
55
109
  version: 1.10.6
56
110
  - !ruby/object:Gem::Dependency
111
+ name: rake
57
112
  requirement: !ruby/object:Gem::Requirement
58
113
  requirements:
59
114
  - - ">="
60
115
  - !ruby/object:Gem::Version
61
116
  version: '10.0'
62
- name: rake
63
- prerelease: false
64
117
  type: :development
118
+ prerelease: false
65
119
  version_requirements: !ruby/object:Gem::Requirement
66
120
  requirements:
67
121
  - - ">="
@@ -83,50 +137,6 @@ files:
83
137
  - README.md
84
138
  - Rakefile
85
139
  - embulk-output-bigquery.gemspec
86
- - example/config_append_direct_schema_update_options.yml
87
- - example/config_client_options.yml
88
- - example/config_csv.yml
89
- - example/config_delete_in_advance.yml
90
- - example/config_delete_in_advance_field_partitioned_table.yml
91
- - example/config_delete_in_advance_partitioned_table.yml
92
- - example/config_expose_errors.yml
93
- - example/config_gcs.yml
94
- - example/config_guess_from_embulk_schema.yml
95
- - example/config_guess_with_column_options.yml
96
- - example/config_gzip.yml
97
- - example/config_jsonl.yml
98
- - example/config_max_threads.yml
99
- - example/config_min_ouput_tasks.yml
100
- - example/config_mode_append.yml
101
- - example/config_mode_append_direct.yml
102
- - example/config_nested_record.yml
103
- - example/config_payload_column.yml
104
- - example/config_payload_column_index.yml
105
- - example/config_progress_log_interval.yml
106
- - example/config_replace.yml
107
- - example/config_replace_backup.yml
108
- - example/config_replace_backup_field_partitioned_table.yml
109
- - example/config_replace_backup_partitioned_table.yml
110
- - example/config_replace_field_partitioned_table.yml
111
- - example/config_replace_partitioned_table.yml
112
- - example/config_replace_schema_update_options.yml
113
- - example/config_skip_file_generation.yml
114
- - example/config_table_strftime.yml
115
- - example/config_template_table.yml
116
- - example/config_uncompressed.yml
117
- - example/config_with_rehearsal.yml
118
- - example/example.csv
119
- - example/example.yml
120
- - example/example2_1.csv
121
- - example/example2_2.csv
122
- - example/example4_1.csv
123
- - example/example4_2.csv
124
- - example/example4_3.csv
125
- - example/example4_4.csv
126
- - example/json_key.json
127
- - example/nested_example.jsonl
128
- - example/schema.json
129
- - example/schema_expose_errors.json
130
140
  - lib/embulk/output/bigquery.rb
131
141
  - lib/embulk/output/bigquery/auth.rb
132
142
  - lib/embulk/output/bigquery/bigquery_client.rb
@@ -147,7 +157,7 @@ homepage: https://github.com/embulk/embulk-output-bigquery
147
157
  licenses:
148
158
  - MIT
149
159
  metadata: {}
150
- post_install_message:
160
+ post_install_message:
151
161
  rdoc_options: []
152
162
  require_paths:
153
163
  - lib
@@ -163,7 +173,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
163
173
  version: '0'
164
174
  requirements: []
165
175
  rubygems_version: 3.0.3
166
- signing_key:
176
+ signing_key:
167
177
  specification_version: 4
168
178
  summary: Google BigQuery output plugin for Embulk
169
179
  test_files: