embulk-input-google_spreadsheets 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: '0298679088554e589d5fcf35d28052321996397b'
+  data.tar.gz: 437d4de2a144529e3a12c734bf2742c475632c4b
+SHA512:
+  metadata.gz: f92a35388a40ac64fe15dde0eab169db9471884b4e27742ee5bb98ed94f994a528f6ad57c0d7400095953ac8864ba7907c600297b5a2d3f2df27096bbbab530e
+  data.tar.gz: 0c391fa5d167d32471d3881a77cc64012fe42a6aafa76f8a13250ef734a87d32d8aab24320db72b6b423ee2056deffc606ce2ad5f8aa9300fd591619b19f64ab

data/.gitignore

@@ -0,0 +1,10 @@
+*~
+/pkg/
+/tmp/
+/.bundle/
+/Gemfile.lock
+vendor/
+bin/
+.ruby-version
+example/authorized_user_credentials.json
+example/service_account_credentials.json

data/.travis.yml

@@ -0,0 +1,12 @@
+language: ruby
+cache: bundler
+rvm:
+  - jruby-9.0.5.0
+  - jruby-head
+jdk:
+  - openjdk7
+before_install:
+  - gem install bundler
+matrix:
+  allow_failures:
+    - rvm: jruby-head

data/CHANGELOG.md

@@ -0,0 +1,67 @@
+1.0.0 (2017-03-27)
+==================
+
+### Now, embulk-input-googlespreadsheet is recreated as embulk-input-google_spreadsheets, so this release has **lots of breaking changes**.
+
+#### About Configuration
+- Remove `account` option because this parameter can be set by `json_keyfile` or internal processing.
+- Change acceptable values of `auth_method` option to `service_account`, `authorized_user`, `compute_engine`, and `application_default`.
+  - `json_key` is renamed to `service_account`.
+  - `refresh_token` is renamed to `authorized_user`.
+- Remove `spreadsheet_key` option because it is ambiguous what to specify for this option.
+- Create `spreadsheets_url` option instead of `spreadsheet_key` option.
+- Create `max_fetch_rows` option for loading a worksheet which have lots of cells.
+  - Load data from a worksheet for each numerical value specified by this option.
+- Create `null_string` option to define `NULL` value.
+- Create `stop_on_invalid_record` option.
+  - Stop loading data if this option is true.
+- Create `default_timestamp_format` option.
+- Create `default_timezone` option.
+- Create `typecast` option for `columns` configurations.
+  - You can choose `strict`, `loose`, `minimal`.
+  - You can set default by `default_typecast` option.
+
+#### About Behaviours
+- Improve memory usage
+  - The previous implementation loads all data (includes other worksheets) from the spreadsheets at once by each task, so embulk uses lots of memory.
+  - This version loads data from the specified worksheet for each numerical value specified by `max_fetch_rows` option.
+- Change default timestamp format `%Y-%m-%d %H:%M:%S` to `%Y-%m-%d %H:%M:%S.%N %z`.
+  - This new default format follows embulk-core implementation.
+    - Ref. https://github.com/embulk/embulk/blob/936c5d5a20af3086f7d1e5779a89035105bb975b/embulk-core/src/main/java/org/embulk/spi/type/TimestampType.java#L10
+    - But, `Time.strptime` does not support `%6N`, so this plugin use `%N` instead.
+  - Enable to define default timestamp format by using `default_timestamp_format` option.
+- Change default timezone from `Asia/Tokyo` to `UTC`.
+  - This follows world standard.
+  - Enable to define default timezone by using `default_timezone` option.
+- Remove mysterious replace of string.
+  - The previous version replace `\t` and `\n` to ` `, but the implementation is so mysterious for users.
+  - This version does not do any mysterious replaces.
+- Change errors more traceable.
+  - All errors are wrapped by `Embulk::Input::GoogleSpreadsheets::ConfigError` or `Embulk::Input::GoogleSpreadsheets::DataError`, so you can see JRuby stacktrace when some error occurs.
+- Improve processing invalid records.
+  - Enable to skip by set `stop_on_invalid_record` option false.
+- Improve Typecasting.
+  - Enable to define `NULL` value by using `null_string` option.
+  - Enable to use `yes` or `no` as boolean type.
+  - Disable to round values.
+    - The previous version typecast empty string to `0` if the type is `long`.
+    - The previous version typecast empty string to `0.0` if the type is `double`.
+  - Support loose typecasting.
+  - Support minimal typecasting.
+  
+#### Others
+- Add tests
+- Rename this plugin's name to `embulk-input-google_spreadsheets`
+- Start CI
+  - https://travis-ci.org/medjed/embulk-input-google_spreadsheets
+  
+#### Known Issues
+- [Some multi-byte strings cannot be used as a worksheet title.](https://github.com/medjed/embulk-input-google_spreadsheets/issues/6)
+- [Emoji cannot be used as a worksheet title.](https://github.com/medjed/embulk-input-google_spreadsheets/issues/7)
+
+0.3.0 (2017-03-08)
+==================
+
+- change: transfer `apollocarlos/embulk-input-googlespreadsheet` to `medjed/embulk-input-googlespreadsheet`
+- incompatible change: raise `ConfigError` if worksheet cannot be found.
+- incompatible change: change `CompatibilityError` to `UnmatchedNumberOfColumnsError` .

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org/'
+
+gemspec

data/LICENSE.txt

@@ -0,0 +1,21 @@
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,94 @@
+# Google Spreadsheets input plugin for Embulk
+
+[![Build Status](https://secure.travis-ci.org/medjed/embulk-input-google_spreadsheets.png?branch=master)](http://travis-ci.org/medjed/embulk-input-google_spreadsheets)
+
+Embulk input plugin to load records from Google Spreadsheets.
+
+## Overview
+
+* **Plugin type**: input
+
+## Configuration
+
+| name                | type        | requirement | default         | description            |
+|:--------------------|:------------|:------------|:----------------|:-----------------------|
+| auth_method        | string      | optional    | `authorized_user`      | `service_account`, `authorized_user`, `compute_engine`, or `application_default` |
+| json_keyfile       | string      | optional    |                 | keyfile path or `content` |
+| spreadsheets_url    | string      | required    |                 |  |
+| worksheet_title    | string      | required    |                 | worksheet title |
+| start_column       | integer     | optional    | `1`             |  |
+| start_row          | integer     | optional    | `1`             |  |
+| end_row            | integer     | optional    | `-1`            | `-1` means loading records until an empty record appears. |
+| max_fetch_rows     | integer     | optional    | `10000`        |  Load data from a worksheet for each numerical value specified by this option. |
+| null_string        | string      | optional    | `''`            |  Replace this value to `NULL` |
+| stop_on_invalid_record | boolean | optional    | `true`          |  |
+| default_timestamp_format | string | optional | `'%Y-%m-%d %H:%M:%S.%N %z'` | |
+| default_timezone | string | optional | `'UTC'` | |
+| default_typecast | string | optional | `'strict'` | |
+|  columns            | array       | required    |                 |  |
+
+##### about keyfile
+* if `auth_method` is `compute_engine` or `application_default`, this option is not required.
+* if `auth_method` is `authorized_user`, this plugin supposes the format is the below.
+  
+  ```json
+  {
+    "client_id":"xxxxxxxxxxx.apps.googleusercontent.com",
+    "client_secret":"xxxxxxxxxxx",
+    "refresh_token":"xxxxxxxxxxx"
+  }
+  ```
+* if `auth_method` is `service_account`, set the service account credential json file path.
+
+##### about columns
+* name: column name
+* type: boolean, long, double, string, timestamp, json
+* format: timestamp format like `'%Y-%m-%d %H:%M:%S.%N %z'`
+* timezone: timezone
+* typecast: you can choose `strict`, `loose`, `minimal` (default: `strict`)
+  * `strict`: raise TypecastError if typecasting is failed.
+  * `loose` : set `NULL` value if typecasting is failed.
+  * `minimal`  : typecast minimally.
+
+
+## Development
+
+### Run example:
+
+1. Execute `example/setup_authorized_user_credentials.rb` if you don't have credentials, then
+2. 
+
+```
+$ embulk bundle install --path vendor/bundle
+$ embulk run -b . -l trace example/config_authorized_user.yml
+```
+
+### Run test:
+
+```
+$ bundle exec rake test
+```
+
+To run tests which actually connects to Google Spreadsheets such as `test/test_google_spreadsheets_client.rb`,
+prepare a json\_keyfile at `example/service_account_credentials.json`, then
+
+```
+$ bundle exec ruby test/test_google_spreadsheets_client.rb
+$ bundle exec ruby test/test_example.rb
+```
+
+### Release gem:
+
+Fix gemspec, then
+
+```
+$ bundle exec rake release
+```
+
+## ChangeLog
+
+[CHANGELOG.md](CHANGELOG.md)
+
+## Known Issues
+
+[Github Issues](https://github.com/medjed/embulk-input-google_spreadsheets/issues)

data/Rakefile

@@ -0,0 +1,13 @@
+require "bundler/gem_tasks"
+
+require 'rake/testtask'
+
+desc 'Run test_unit based test'
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.test_files = Dir["test/**/test_*.rb"].sort
+  t.verbose = true
+  t.warning = false
+end
+task :default => :test
+

data/embulk-input-google_spreadsheets.gemspec

@@ -0,0 +1,24 @@
+Gem::Specification.new do |spec|
+  spec.name          = 'embulk-input-google_spreadsheets'
+  spec.version       = '1.0.0'
+  spec.authors       = %w(Civitaspo yang-xu)
+  spec.summary       = 'Google Spreadsheets input plugin for Embulk'
+  spec.description   = 'Load records from Google Spreadsheets.'
+  spec.email         = %w(civitaspo@gmail.com xu.yang.9.65@gmail.com)
+  spec.licenses      = ['MIT']
+  spec.homepage      = 'https://github.com/medjed/embulk-input-google_spreadsheets'
+
+  spec.files         = `git ls-files`.split("\n") + Dir['classpath/*.jar']
+  spec.test_files    = spec.files.grep(%r{^(test|spec)/})
+  spec.require_paths = ['lib']
+
+  spec.add_development_dependency 'embulk', '>= 0.8.1'
+  spec.add_development_dependency 'bundler'
+  spec.add_development_dependency 'rake'
+  spec.add_development_dependency 'highline'
+  spec.add_development_dependency 'test-unit'
+  spec.add_development_dependency 'test-unit-rr'
+
+  spec.add_dependency 'google-api-client', '>= 0.9.15'
+  spec.add_dependency 'time_with_zone'
+end

data/example/config_authorized_user.yml

@@ -0,0 +1,19 @@
+in:
+  type: google_spreadsheets
+  auth_method: authorized_user
+  json_keyfile: example/authorized_user_credentials.json
+  spreadsheets_url: https://docs.google.com/spreadsheets/d/1Cxz-LudQuhRAGZL8mBoHs6mRnpjODpyF4Rwc5UYoV1E/edit#gid=0
+  worksheet_title: sheet1
+  start_row: 2
+  default_timezone: Asia/Tokyo
+  null_string: '\N'
+  default_typecast: strict
+  columns:
+    - {name: _c1, type: boolean}
+    - {name: _c2, type: string}
+    - {name: _c3, type: long}
+    - {name: _c4, type: double}
+    - {name: _c5, type: timestamp, format: '%Y-%m-%d %H:%M:%S.%N'}
+    - {name: _c6, type: timestamp, format: '%Y-%m-%d'}
+out:
+  type: stdout

data/example/config_authorized_user.yml.liquid

@@ -0,0 +1,25 @@
+in:
+  type: google_spreadsheets
+  auth_method: authorized_user
+  json_keyfile:
+    content: |
+      {
+        "client_id": "{{ env.GOOGLE_AUTH_CLIENT_ID }}",
+        "client_secret": "{{ env.GOOGLE_AUTH_CLIENT_SECRET }}",
+        "refresh_token": "{{ env.GOOGLE_AUTH_REFRESH_TOKEN }}"
+      }
+  spreadsheets_url: https://docs.google.com/spreadsheets/d/1Cxz-LudQuhRAGZL8mBoHs6mRnpjODpyF4Rwc5UYoV1E/edit#gid=0
+  worksheet_title: sheet1
+  start_column: 2
+  default_timezone: Asia/Tokyo
+  null_string: '\N'
+  columns:
+    - {name: _c1, type: boolean}
+    - {name: _c2, type: string}
+    - {name: _c3, type: long}
+    - {name: _c4, type: double}
+    - {name: _c5, type: timestamp, format: '%Y-%m-%d %H:%M:%S.%N'}
+    - {name: _c6, type: timestamp, format: '%Y-%m-%d'}
+
+out:
+  type: stdout

data/example/config_authorized_user_emoji_worksheet.yml

@@ -0,0 +1,19 @@
+in:
+  type: google_spreadsheets
+  auth_method: authorized_user
+  json_keyfile: example/authorized_user_credentials.json
+  spreadsheets_url: https://docs.google.com/spreadsheets/d/1Cxz-LudQuhRAGZL8mBoHs6mRnpjODpyF4Rwc5UYoV1E/edit#gid=0
+  worksheet_title: 🍣emoji🍣
+  start_row: 2
+  default_timezone: Asia/Tokyo
+  null_string: '\N'
+  default_typecast: strict
+  columns:
+    - {name: _c1, type: boolean}
+    - {name: _c2, type: string}
+    - {name: _c3, type: long}
+    - {name: _c4, type: double}
+    - {name: _c5, type: timestamp, format: '%Y-%m-%d %H:%M:%S.%N'}
+    - {name: _c6, type: timestamp, format: '%Y-%m-%d'}
+out:
+  type: stdout

data/example/config_authorized_user_empty_rows_appears_at_the_same_as_max_fetch_rows.yml

@@ -0,0 +1,20 @@
+in:
+  type: google_spreadsheets
+  auth_method: authorized_user
+  json_keyfile: example/authorized_user_credentials.json
+  spreadsheets_url: https://docs.google.com/spreadsheets/d/1Cxz-LudQuhRAGZL8mBoHs6mRnpjODpyF4Rwc5UYoV1E/edit#gid=0
+  worksheet_title: Empty rows appears at the same as max_fetch_rows
+  start_row: 2
+  max_fetch_rows: 2
+  default_timezone: Asia/Tokyo
+  null_string: '\N'
+  default_typecast: strict
+  columns:
+    - {name: _c1, type: boolean}
+    - {name: _c2, type: string}
+    - {name: _c3, type: long}
+    - {name: _c4, type: double}
+    - {name: _c5, type: timestamp, format: '%Y-%m-%d %H:%M:%S.%N'}
+    - {name: _c6, type: timestamp, format: '%Y-%m-%d'}
+out:
+  type: stdout

data/example/config_authorized_user_large_data.yml

@@ -0,0 +1,19 @@
+in:
+  type: google_spreadsheets
+  auth_method: authorized_user
+  json_keyfile: example/authorized_user_credentials.json
+  spreadsheets_url: https://docs.google.com/spreadsheets/d/1Cxz-LudQuhRAGZL8mBoHs6mRnpjODpyF4Rwc5UYoV1E/edit#gid=0
+  worksheet_title: Large Data
+  start_row: 2
+  default_timezone: Asia/Tokyo
+  null_string: '\N'
+  default_typecast: strict
+  columns:
+    - {name: _c1, type: boolean}
+    - {name: _c2, type: string}
+    - {name: _c3, type: long}
+    - {name: _c4, type: double}
+    - {name: _c5, type: timestamp, format: '%Y-%m-%d %H:%M:%S.%N'}
+    - {name: _c6, type: timestamp, format: '%Y-%m-%d'}
+out:
+  type: stdout

data/example/config_authorized_user_no_data.yml

@@ -0,0 +1,18 @@
+in:
+  type: google_spreadsheets
+  auth_method: authorized_user
+  json_keyfile: example/authorized_user_credentials.json
+  spreadsheets_url: https://docs.google.com/spreadsheets/d/1Cxz-LudQuhRAGZL8mBoHs6mRnpjODpyF4Rwc5UYoV1E/edit#gid=0
+  worksheet_title: No Data
+  default_timezone: Asia/Tokyo
+  null_string: '\N'
+  default_typecast: strict
+  columns:
+    - {name: _c1, type: boolean}
+    - {name: _c2, type: string}
+    - {name: _c3, type: long}
+    - {name: _c4, type: double}
+    - {name: _c5, type: timestamp, format: '%Y-%m-%d %H:%M:%S.%N'}
+    - {name: _c6, type: timestamp, format: '%Y-%m-%d'}
+out:
+  type: stdout

data/example/config_service_account.yml

@@ -0,0 +1,19 @@
+in:
+  type: google_spreadsheets
+  auth_method: service_account
+  json_keyfile: example/service_account_credentials.json
+  spreadsheets_url: https://docs.google.com/spreadsheets/d/1Cxz-LudQuhRAGZL8mBoHs6mRnpjODpyF4Rwc5UYoV1E/edit#gid=0
+  worksheet_title: sheet1
+  start_row: 2
+  default_timezone: Asia/Tokyo
+  null_string: '\N'
+  default_typecast: strict
+  columns:
+    - {name: _c1, type: boolean}
+    - {name: _c2, type: string}
+    - {name: _c3, type: long}
+    - {name: _c4, type: double}
+    - {name: _c5, type: timestamp, format: '%Y-%m-%d %H:%M:%S.%N'}
+    - {name: _c6, type: timestamp, format: '%Y-%m-%d'}
+out:
+  type: stdout

data/example/config_service_account_emoji_worksheet.yml

@@ -0,0 +1,19 @@
+in:
+  type: google_spreadsheets
+  auth_method: service_account
+  json_keyfile: example/service_account_credentials.json
+  spreadsheets_url: https://docs.google.com/spreadsheets/d/1Cxz-LudQuhRAGZL8mBoHs6mRnpjODpyF4Rwc5UYoV1E/edit#gid=0
+  worksheet_title: 🍣emoji🍣
+  start_row: 2
+  default_timezone: Asia/Tokyo
+  null_string: '\N'
+  default_typecast: strict
+  columns:
+    - {name: _c1, type: boolean}
+    - {name: _c2, type: string}
+    - {name: _c3, type: long}
+    - {name: _c4, type: double}
+    - {name: _c5, type: timestamp, format: '%Y-%m-%d %H:%M:%S.%N'}
+    - {name: _c6, type: timestamp, format: '%Y-%m-%d'}
+out:
+  type: stdout

data/example/config_service_account_empty_rows_appears_at_the_same_as_max_fetch_rows.yml

@@ -0,0 +1,20 @@
+in:
+  type: google_spreadsheets
+  auth_method: service_account
+  json_keyfile: example/service_account_credentials.json
+  spreadsheets_url: https://docs.google.com/spreadsheets/d/1Cxz-LudQuhRAGZL8mBoHs6mRnpjODpyF4Rwc5UYoV1E/edit#gid=0
+  worksheet_title: Empty rows appears at the same as max_fetch_rows
+  start_row: 2
+  max_fetch_rows: 2
+  default_timezone: Asia/Tokyo
+  null_string: '\N'
+  default_typecast: strict
+  columns:
+    - {name: _c1, type: boolean}
+    - {name: _c2, type: string}
+    - {name: _c3, type: long}
+    - {name: _c4, type: double}
+    - {name: _c5, type: timestamp, format: '%Y-%m-%d %H:%M:%S.%N'}
+    - {name: _c6, type: timestamp, format: '%Y-%m-%d'}
+out:
+  type: stdout