embulk-output-bigquery 0.5.0 → 0.6.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA256:
-  metadata.gz: 3e0087103039718cb24224b6bb793d820b53b935194d412e4b2984aba3d7d7a8
-  data.tar.gz: 9ac27a3b881277450cbfaa096de0690c721a8f86f0e78abb692c8a4ed5b679d5
+SHA1:
+  metadata.gz: 7ea0cf04e91a092e3d97bf3e46d1c181aab4943f
+  data.tar.gz: 60e970acbc16128189df8c274a832d23160e4c80
 SHA512:
-  metadata.gz: 6b0ccf4e349a5d15321cfcc97138a98676bddfd412fd6fadfc8b1e0d6cd31d9739a8a5f46ccd923644543ae43cc0134b3e7598f80d89c330a4ac8aec49c084c1
-  data.tar.gz: f02557cdd7956620ae59eb6bc0e5872992d20a65881bd69230b0b0442342a36203d1eedd8a20702d2000f412b909359657bfa300b3e82b5f494398ea6e5ea301
+  metadata.gz: 14fb288ad9781515a28cf72869ca7e76081202b8cf7e29d2448bb9ec37e6a8e8e73a046930f53c41785551b354a990ccc8ee2f9298b418de672c6dfaa2e6447b
+  data.tar.gz: 9f8f59c89cf7cc9974ab8a287ffbf522263ff5e81ff988de95f22c9086cf805d39d8f9c7c7d41310357464a29e5ec0404fe819b5ba41de36eb0d870c4ca31144

data/CHANGELOG.md

@@ -1,9 +1,34 @@
+## 0.6.4 - 2019-11-06
+
+* [enhancement] Add DATETIME type conveter (thanks to @kekekenta)
+
+## 0.6.3 - 2019-10-28
+
+* [enhancement] Add DATE type conveter (thanks to @tksfjt1024)
+
+## 0.6.2 - 2019-10-16
+
+* [maintenance] Lock signet and google-api-client version (thanks to @hiroyuki-sato)
+
+## 0.6.1 - 2019-08-28
+
+* [maintenance] Release a new gem not to include symlinks to make it work on Windows.
+
+## 0.6.0 - 2019-08-11
+
+Cleanup `auth_method`:
+
+* [enhancement] Support `auth_method: authorized_user` (OAuth)
+* [incompatibility change] Rename `auth_method: json_key` to `auth_method: service_account` (`json_key` is kept for backward compatibility)
+* [incompatibility change] Remove deprecated `auth_method: private_key` (p12 key)
+* [incompatibility change] Change the default `auth_method` to `application_default` from `private_key` because `private_key` was dropped.
+
 ## 0.5.0 - 2019-08-10
 
-* [incompatibility change] Drop deprecated time\_partitioning.require\_partition\_filter
-* [incompatibility change] Drop prevent\_duplicate\_insert which has no use-case now
-* [incompatibility change] Change default value of `auto\_create\_table` to `true` from `false`
-  * Modes `replace`, `replace_backup`, `append`, `delete_in_advance`, that is, except `append_direct` requires `auto_create_table: true`.
+* [incompatibility change] Drop deprecated `time_partitioning`.`require_partition_filter`
+* [incompatibility change] Drop `prevent_duplicate_insert` which has no use-case now
+* [incompatibility change] Modes `replace`, `replace_backup`, `append`, and `delete_in_advance` require `auto_create_table: true` now because, previously, these modes had created a target table even with `auto_create_table: false` and made users being confused. Note that `auto_create_table: true` is always required even for a partition (a table name with a partition decorator) which may not require creating a table. This is for simplicity of logics and implementations.
+* [incompatibility change] Change default value of `auto_create_table` to `true` because the above 4 modes, that is, except `append_direct` always require `auto_create_table: true` now.
 
 ## 0.4.14 - 2019-08-10
 

data/README.md

@@ -29,17 +29,15 @@ OAuth flow for installed applications.
 
 | name                                 | type        | required?  | default                  | description            |
 |:-------------------------------------|:------------|:-----------|:-------------------------|:-----------------------|
-|  mode                                | string      | optional   | "append"                 | See [Mode](#mode)     |
-|  auth_method                         | string      | optional   | "private_key"            | `private_key` , `json_key` or `compute_engine`
-|  service_account_email               | string      | required when auth_method is private_key  |   | Your Google service account email
-|  p12_keyfile                         | string      | required when auth_method is private_key  |   | Fullpath of private key in P12(PKCS12) format |
-|  json_keyfile                        | string      | required when auth_method is json_key     |   | Fullpath of json key |
-|  project                             | string      | required if json_keyfile is not given     |   | project_id |
+|  mode                                | string      | optional   | "append"                 | See [Mode](#mode)      |
+|  auth_method                         | string      | optional   | "application\_default"   | See [Authentication](#authentication) |
+|  json_keyfile                        | string      | optional   |                          | keyfile path or `content` |
+|  project                             | string      | required unless service\_account's `json_keyfile` is given. | | project\_id |
 |  dataset                             | string      | required   |                          | dataset |
 |  location                            | string      | optional   | nil                      | geographic location of dataset. See [Location](#location) |
 |  table                               | string      | required   |                          | table name, or table name with a partition decorator such as `table_name$20160929`|
 |  auto_create_dataset                 | boolean     | optional   | false                    | automatically create dataset |
-|  auto_create_table                   | boolean     | optional   | true                     | `false` is available only for `append_direct` mode. Other modes requires `true`. See [Dynamic Table Creating](#dynamic-table-creating) and [Time Partitioning](#time-partitioning) |
+|  auto_create_table                   | boolean     | optional   | true                     | `false` is available only for `append_direct` mode. Other modes require `true`. See [Dynamic Table Creating](#dynamic-table-creating) and [Time Partitioning](#time-partitioning) |
 |  schema_file                         | string      | optional   |                          | /path/to/schema.json |
 |  template_table                      | string      | optional   |                          | template table name. See [Dynamic Table Creating](#dynamic-table-creating) |
 |  job_status_max_polling_time         | int         | optional   | 3600 sec                 | Max job status polling time |
@@ -108,9 +106,8 @@ Following options are same as [bq command-line tools](https://cloud.google.com/b
 out:
   type: bigquery
   mode: append
-  auth_method: private_key   # default
-  service_account_email: ABCXYZ123ABCXYZ123.gserviceaccount.com
-  p12_keyfile: /path/to/p12_keyfile.p12
+  auth_method: service_account
+  json_keyfile: /path/to/json_keyfile.json
   project: your-project-000
   dataset: your_dataset_name
   table: your_table_name
@@ -118,7 +115,7 @@ out:
   source_format: NEWLINE_DELIMITED_JSON
 ```
 
-### location
+### Location
 
 The geographic location of the dataset. Required except for US and EU.
 
@@ -126,7 +123,7 @@ GCS bucket should be in same region when you use `gcs_bucket`.
 
 See also [Dataset Locations | BigQuery | Google Cloud](https://cloud.google.com/bigquery/docs/dataset-locations)
 
-### mode
+### Mode
 
 5 modes are provided.
 
@@ -165,53 +162,69 @@ NOTE: BigQuery does not support replacing (actually, copying into) a non-partiti
 
 ### Authentication
 
-There are three methods supported to fetch access token for the service account.
+There are four authentication methods
+
+1. `service_account` (or `json_key` for backward compatibility)
+1. `authorized_user`
+1. `compute_engine`
+1. `application_default`
 
-1. Public-Private key pair of GCP(Google Cloud Platform)'s service account
-2. JSON key of GCP(Google Cloud Platform)'s service account
-3. Pre-defined access token (Google Compute Engine only)
+#### service\_account (or json\_key)
 
-#### Public-Private key pair of GCP's service account
+Use GCP service account credentials.
+You first need to create a service account, download its json key and deploy the key with embulk.
+
+```yaml
+out:
+  type: bigquery
+  auth_method: service_account
+  json_keyfile: /path/to/json_keyfile.json
+```
 
-You first need to create a service account (client ID),
-download its private key and deploy the key with embulk.
+You can also embed contents of `json_keyfile` at config.yml.
 
 ```yaml
 out:
   type: bigquery
-  auth_method: private_key   # default
-  service_account_email: ABCXYZ123ABCXYZ123.gserviceaccount.com
-  p12_keyfile: /path/to/p12_keyfile.p12
+  auth_method: service_account
+  json_keyfile:
+    content: |
+      {
+          "private_key_id": "123456789",
+          "private_key": "-----BEGIN PRIVATE KEY-----\nABCDEF",
+          "client_email": "..."
+      }
 ```
 
-#### JSON key of GCP's service account
+#### authorized\_user
 
-You first need to create a service account (client ID),
-download its json key and deploy the key with embulk.
+Use Google user credentials.
+You can get your credentials at `~/.config/gcloud/application_default_credentials.json` by running `gcloud auth login`.
 
 ```yaml
 out:
   type: bigquery
-  auth_method: json_key
-  json_keyfile: /path/to/json_keyfile.json
+  auth_method: authorized_user
+  json_keyfile: /path/to/credentials.json
 ```
 
-You can also embed contents of json_keyfile at config.yml.
+You can also embed contents of `json_keyfile` at config.yml.
 
 ```yaml
 out:
   type: bigquery
-  auth_method: json_key
+  auth_method: authorized_user
   json_keyfile:
     content: |
       {
-          "private_key_id": "123456789",
-          "private_key": "-----BEGIN PRIVATE KEY-----\nABCDEF",
-          "client_email": "..."
-       }
+        "client_id":"xxxxxxxxxxx.apps.googleusercontent.com",
+        "client_secret":"xxxxxxxxxxx",
+        "refresh_token":"xxxxxxxxxxx",
+        "type":"authorized_user"
+      }
 ```
 
-#### Pre-defined access token(GCE only)
+#### compute\_engine
 
 On the other hand, you don't need to explicitly create a service account for embulk when you
 run embulk in Google Compute Engine. In this third authentication method, you need to
@@ -224,6 +237,22 @@ out:
   auth_method: compute_engine
 ```
 
+#### application\_default
+
+Use Application Default Credentials (ADC).  ADC is a strategy to locate Google Cloud Service Account credentials.
+
+1. ADC checks to see if the environment variable `GOOGLE_APPLICATION_CREDENTIALS` is set. If the variable is set, ADC uses the service account file that the variable points to.
+2. ADC checks to see if `~/.config/gcloud/application_default_credentials.json` is located. This file is created by running `gcloud auth application-default login`.
+3. Use the default service account for credentials if the application running on Compute Engine, App Engine, Kubernetes Engine, Cloud Functions or Cloud Run.
+
+See https://cloud.google.com/docs/authentication/production for details.
+
+```yaml
+out:
+  type: bigquery
+  auth_method: application_default
+```
+
 ### Table id formatting
 
 `table` and option accept [Time#strftime](http://ruby-doc.org/core-1.9.3/Time.html#method-i-strftime)
@@ -232,12 +261,12 @@ Table ids are formatted at runtime
 using the local time of the embulk server.
 
 For example, with the configuration below,
-data is inserted into tables `table_2015_04`, `table_2015_05` and so on.
+data is inserted into tables `table_20150503`, `table_20150504` and so on.
 
 ```yaml
 out:
   type: bigquery
-  table: table_%Y_%m
+  table: table_%Y%m%d
 ```
 
 ### Dynamic table creating
@@ -252,7 +281,7 @@ Please set file path of schema.json.
 out:
   type: bigquery
   auto_create_table: true
-  table: table_%Y_%m
+  table: table_%Y%m%d
   schema_file: /path/to/schema.json
 ```
 
@@ -264,7 +293,7 @@ Plugin will try to read schema from existing table and use it as schema template
 out:
   type: bigquery
   auto_create_table: true
-  table: table_%Y_%m
+  table: table_%Y%m%d
   template_table: existing_table_name
 ```
 
@@ -278,17 +307,17 @@ Column options are used to aid guessing BigQuery schema, or to define conversion
 
 - **column_options**: advanced: an array of options for columns
   - **name**: column name
-  - **type**: BigQuery type such as `BOOLEAN`, `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP`, and `RECORD`. See belows for supported conversion type.
+  - **type**: BigQuery type such as `BOOLEAN`, `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP`, `DATETIME`, `DATE`, and `RECORD`. See belows for supported conversion type.
     - boolean:   `BOOLEAN`, `STRING` (default: `BOOLEAN`)
     - long:      `BOOLEAN`, `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP` (default: `INTEGER`)
     - double:    `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP` (default: `FLOAT`)
-    - string:    `BOOLEAN`, `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP`, `RECORD` (default: `STRING`)
-    - timestamp: `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP` (default: `TIMESTAMP`)
+    - string:    `BOOLEAN`, `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP`, `DATETIME`, `DATE`, `RECORD` (default: `STRING`)
+    - timestamp: `INTEGER`, `FLOAT`, `STRING`, `TIMESTAMP`, `DATETIME`, `DATE` (default: `TIMESTAMP`)
     - json:      `STRING`,  `RECORD` (default: `STRING`)
   - **mode**: BigQuery mode such as `NULLABLE`, `REQUIRED`, and `REPEATED` (string, default: `NULLABLE`)
   - **fields**: Describes the nested schema fields if the type property is set to RECORD. Please note that this is **required** for `RECORD` column.
   - **timestamp_format**: timestamp format to convert into/from `timestamp` (string, default is `default_timestamp_format`)
-  - **timezone**: timezone to convert into/from `timestamp` (string, default is `default_timezone`).
+  - **timezone**: timezone to convert into/from `timestamp`, `date` (string, default is `default_timezone`).
 - **default_timestamp_format**: default timestamp format for column_options (string, default is "%Y-%m-%d %H:%M:%S.%6N")
 - **default_timezone**: default timezone for column_options (string, default is "UTC")
 

data/embulk-output-bigquery.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |spec|
   spec.name          = "embulk-output-bigquery"
-  spec.version       = "0.5.0"
+  spec.version       = "0.6.4"
   spec.authors       = ["Satoshi Akama", "Naotoshi Seo"]
   spec.summary       = "Google BigQuery output plugin for Embulk"
   spec.description   = "Embulk plugin that insert records to Google BigQuery."
@@ -8,11 +8,18 @@ Gem::Specification.new do |spec|
   spec.licenses      = ["MIT"]
   spec.homepage      = "https://github.com/embulk/embulk-output-bigquery"
 
-  spec.files         = `git ls-files`.split("\n") + Dir["classpath/*.jar"]
+  # Exclude example directory which uses symlinks from generating gem.
+  # Symlinks do not work properly on the Windows platform without administrator privilege.
+  spec.files         = `git ls-files`.split("\n") + Dir["classpath/*.jar"] - Dir["example/*" ]
   spec.test_files    = spec.files.grep(%r{^(test|spec)/})
   spec.require_paths = ["lib"]
 
-  spec.add_dependency 'google-api-client'
+  # TODO
+  # signet 0.12.0 and google-api-client 0.33.0 require >= Ruby 2.4.
+  # Embulk 0.9 use JRuby 9.1.X.Y and It compatible Ruby 2.3.
+  # So, Force install signet < 0.12 and google-api-client < 0.33.0
+  spec.add_dependency 'signet', '~> 0.7', '< 0.12.0'
+  spec.add_dependency 'google-api-client','< 0.33.0'
   spec.add_dependency 'time_with_zone'
 
   spec.add_development_dependency 'bundler', ['>= 1.10.6']

data/lib/embulk/output/bigquery.rb

@@ -23,7 +23,7 @@ module Embulk
         # @return JSON string
         def self.load(v)
           if v.is_a?(String) # path
-            File.read(v)
+            File.read(File.expand_path(v))
           elsif v.is_a?(Hash)
             v['content']
           end
@@ -33,9 +33,7 @@ module Embulk
       def self.configure(config, schema, task_count)
         task = {
           'mode'                           => config.param('mode',                           :string,  :default => 'append'),
-          'auth_method'                    => config.param('auth_method',                    :string,  :default => 'private_key'),
-          'service_account_email'          => config.param('service_account_email',          :string,  :default => nil),
-          'p12_keyfile'                    => config.param('p12_keyfile',                    :string,  :default => nil),
+          'auth_method'                    => config.param('auth_method',                    :string,  :default => 'application_default'),
           'json_keyfile'                   => config.param('json_keyfile',                  LocalFile, :default => nil),
           'project'                        => config.param('project',                        :string,  :default => nil),
           'dataset'                        => config.param('dataset',                        :string),
@@ -125,28 +123,21 @@ module Embulk
         end
 
         task['auth_method'] = task['auth_method'].downcase
-        unless %w[private_key json_key compute_engine application_default].include?(task['auth_method'])
-          raise ConfigError.new "`auth_method` must be one of private_key, json_key, compute_engine, application_default"
+        unless %w[json_key service_account authorized_user compute_engine application_default].include?(task['auth_method'])
+          raise ConfigError.new "`auth_method` must be one of service_account (or json_key), authorized_user, compute_engine, application_default"
         end
-        if task['auth_method'] == 'private_key' and task['p12_keyfile'].nil?
-          raise ConfigError.new "`p12_keyfile` is required for auth_method private_key"
-        end
-        if task['auth_method'] == 'json_key' and task['json_keyfile'].nil?
-          raise ConfigError.new "`json_keyfile` is required for auth_method json_key"
+        if (task['auth_method'] == 'service_account' or task['auth_method'] == 'json_key') and task['json_keyfile'].nil?
+          raise ConfigError.new "`json_keyfile` is required for auth_method: service_account (or json_key)"
         end
 
-        jsonkey_params = nil
         if task['json_keyfile']
           begin
-            jsonkey_params = JSON.parse(task['json_keyfile'])
+            json_key = JSON.parse(task['json_keyfile'])
+            task['project'] ||= json_key['project_id']
           rescue => e
             raise ConfigError.new "json_keyfile is not a JSON file"
           end
         end
-
-        if jsonkey_params
-          task['project'] ||= jsonkey_params['project_id']
-        end
         if task['project'].nil?
           raise ConfigError.new "Required field \"project\" is not set"
         end
@@ -313,14 +304,14 @@ module Embulk
           bigquery.create_table_if_not_exists(task['table'])
         when 'replace'
           bigquery.create_table_if_not_exists(task['temp_table'])
-          bigquery.create_table_if_not_exists(task['table'])
+          bigquery.create_table_if_not_exists(task['table']) # needs for when task['table'] is a partition
         when 'append'
           bigquery.create_table_if_not_exists(task['temp_table'])
-          bigquery.create_table_if_not_exists(task['table'])
+          bigquery.create_table_if_not_exists(task['table']) # needs for when task['table'] is a partition
         when 'replace_backup'
           bigquery.create_table_if_not_exists(task['temp_table'])
           bigquery.create_table_if_not_exists(task['table'])
-          bigquery.create_table_if_not_exists(task['table_old'], dataset: task['dataset_old'])
+          bigquery.create_table_if_not_exists(task['table_old'], dataset: task['dataset_old']) # needs for when a partition
         else # append_direct
           if task['auto_create_table']
             bigquery.create_table_if_not_exists(task['table'])

data/lib/embulk/output/bigquery/auth.rb

@@ -0,0 +1,35 @@
+require 'googleauth'
+
+module Embulk
+  module Output
+    class Bigquery < OutputPlugin
+      class Auth
+
+        attr_reader :auth_method, :json_key, :scope
+
+        def initialize(task, scope)
+          @auth_method = task['auth_method']
+          @json_key = task['json_keyfile']
+          @scope = scope
+        end
+
+        def authenticate
+          case auth_method
+          when 'authorized_user'
+            key = StringIO.new(json_key)
+            return Google::Auth::UserRefreshCredentials.make_creds(json_key_io: key, scope: scope)
+          when 'compute_engine'
+            return Google::Auth::GCECredentials.new
+          when 'service_account', 'json_key' # json_key is for backward compatibility
+            key = StringIO.new(json_key)
+            return Google::Auth::ServiceAccountCredentials.make_creds(json_key_io: key, scope: scope)
+          when 'application_default'
+            return Google::Auth.get_application_default([scope])
+          else
+            raise ConfigError.new("Unknown auth method: #{auth_method}")
+          end
+        end
+      end
+    end
+  end
+end

data/lib/embulk/output/bigquery/google_client.rb

@@ -1,4 +1,4 @@
-require 'google/api_client/auth/key_utils'
+require_relative 'auth'
 
 module Embulk
   module Output
@@ -14,6 +14,7 @@ module Embulk
         def initialize(task, scope, client_class)
           @task = task
           @scope = scope
+          @auth = Auth.new(task, scope)
           @client_class = client_class
         end
 
@@ -37,39 +38,7 @@ module Embulk
           Embulk.logger.debug { "embulk-output-bigquery: client_options: #{client.client_options.to_h}" }
           Embulk.logger.debug { "embulk-output-bigquery: request_options: #{client.request_options.to_h}" }
 
-          case @task['auth_method']
-          when 'private_key'
-            private_key_passphrase = 'notasecret'
-            key = Google::APIClient::KeyUtils.load_from_pkcs12(@task['p12_keyfile'], private_key_passphrase)
-            auth = Signet::OAuth2::Client.new(
-              token_credential_uri: "https://accounts.google.com/o/oauth2/token",
-              audience: "https://accounts.google.com/o/oauth2/token",
-              scope: @scope,
-              issuer: @task['service_account_email'],
-              signing_key: key)
-
-          when 'compute_engine'
-            auth = Google::Auth::GCECredentials.new
-
-          when 'json_key'
-            json_key = @task['json_keyfile']
-            if File.exist?(json_key)
-              auth = File.open(json_key) do |f|
-                Google::Auth::ServiceAccountCredentials.make_creds(json_key_io: f, scope: @scope)
-              end
-            else
-              key = StringIO.new(json_key)
-              auth = Google::Auth::ServiceAccountCredentials.make_creds(json_key_io: key, scope: @scope)
-            end
-
-          when 'application_default'
-            auth = Google::Auth.get_application_default([@scope])
-
-          else
-            raise ConfigError, "Unknown auth method: #{@task['auth_method']}"
-          end
-
-          client.authorization = auth
+          client.authorization = @auth.authenticate
 
           @cached_client_expiration = Time.now + 1800
           @cached_client = client