embulk-output-bigquery 0.2.0 → 0.2.1

Sign up to get free protection for your applications and to get access to all the features.
Files changed (13) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +55 -0
  3. data/README.md +6 -5
  4. data/build.gradle +21 -3
  5. data/config/checkstyle/checkstyle.xml +128 -0
  6. data/config/checkstyle/default.xml +108 -0
  7. data/gradle/wrapper/gradle-wrapper.jar +0 -0
  8. data/gradle/wrapper/gradle-wrapper.properties +2 -2
  9. data/src/main/java/org/embulk/output/BigqueryAuthentication.java +5 -3
  10. data/src/main/java/org/embulk/output/BigqueryOutputPlugin.java +52 -21
  11. data/src/main/java/org/embulk/output/BigqueryWriter.java +36 -18
  12. metadata +6 -4
  13. data/ChangeLog +0 -50
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 9e4b4a5467fb4589a3d88541af95ee5b002a5f9d
4
- data.tar.gz: 2845c3d881b88183bdf3355647fbfecd6e455372
3
+ metadata.gz: 7e6de19fdc976904bd2ac46aa529e030f9ecbe48
4
+ data.tar.gz: d279269a8914553203a1d2bc06832ed4201bedf3
5
5
  SHA512:
6
- metadata.gz: 61e5481dd18b750555f3462d680cb2b4d680ce768683780e4b2ce2cac6e66134c7eb38227adf8e48a4655447e27e166f14487c129fe9154a164131a4be06270b
7
- data.tar.gz: 09036cb1d4b20556cb36e11839c7807b00ae453976b4192f9c6f489284aa5f1cbe48b6c42e5b9149b1ed3c3d04a92ee9546cce030b3d9e0c0929f70e365d3ae7
6
+ metadata.gz: 3e7f86e4b963012e0a55227b53262c141209d5aaed5c06b82d1d3da844f467c2b0e53c750e31eec38963459e58e02ba54b82fbb39c14084df241428b5245a220
7
+ data.tar.gz: 60850b5ce34b7b626cf9766a537196d160ae1c1bd0365f119817f50977f7a171762e6562ac987cdc2655fc75892dd6beafceebd47305818e625330aaabb1911b
data/CHANGELOG.md ADDED
@@ -0,0 +1,55 @@
1
+ ## 0.2.1 - 2016-01-28
2
+
3
+ * [maintenance] Upgraded Embulk version to 0.8.1 [#22](https://github.com/embulk/embulk-output-bigquery/pull/22). @joker1007 thanks!
4
+ * [maintenance] Formatted code style by checkstyle [#23](https://github.com/embulk/embulk-output-bigquery/pull/23)
5
+
6
+ ## 0.2.0 - 2016-01-26
7
+
8
+ * [new feature] Added mode parameters and support 4 modes(append, replace, replace_backup, delete_in_advance). [#20](https://github.com/embulk/embulk-output-bigquery/pull/20) [#21](https://github.com/embulk/embulk-output-bigquery/pull/21) @joker1007 thanks!
9
+
10
+ ## 0.1.11 - 2015-11-16
11
+
12
+ * [maintenance] Change error result display for easy investigation. [#18](https://github.com/embulk/embulk-output-bigquery/pull/18)
13
+
14
+ ## 0.1.10 - 2015-10-06
15
+
16
+ * [new feature] Added new auth method - json_keyfile of GCP(Google Cloud Platform)'s service account [#17](https://github.com/embulk/embulk-output-bigquery/pull/17)
17
+
18
+ ## 0.1.9 - 2015-08-19
19
+
20
+ * [maintenance] Upgraded Embulk version to 0.7.1
21
+
22
+ ## 0.1.8 - 2015-08-19
23
+
24
+ * [new feature] Supported mapreduce-executor. @frsyuki thanks! [#13](https://github.com/embulk/embulk-output-bigquery/pull/13)
25
+ * [maintenance] Fixed job_id generation logic [#15](https://github.com/embulk/embulk-output-bigquery/pull/15)
26
+ * [maintenance] Refactored [#11](https://github.com/embulk/embulk-output-bigquery/pull/11)
27
+
28
+ ## 0.1.7 - 2015-05-20
29
+
30
+ * [new feature] Added allow_quoted_newlines option [#10](https://github.com/embulk/embulk-output-bigquery/pull/10)
31
+ * [maintenance] Upgraded embulk version to 0.6.8
32
+
33
+ ## 0.1.6 - 2015-04-23
34
+
35
+ * [new feature] Added ignore_unknown_values option to job_id generation logic. [#9](https://github.com/embulk/embulk-output-bigquery/pull/9)
36
+
37
+ ## 0.1.5 - 2015-04-23
38
+
39
+ * [new feature] Added ignore_unknown_values option. [#8](https://github.com/embulk/embulk-output-bigquery/pull/8) @takus thanks!
40
+
41
+ ## 0.1.4 - 2015-04-21
42
+
43
+ * [new feature] Added prevent_duplicate_insert option
44
+
45
+ ## 0.1.3 - 2015-04-06
46
+
47
+ * [new feature] Added new auth method - pre-defined access token of GCE(Google Compute Engine)
48
+ * [maintenance] Updated Google provided libraries
49
+ * http-client:google-http-client-jackson2 from 1.19.0 to 1.20.0
50
+ * apis:google-api-services-bigquery from v2-rev193-1.19.1 to v2-rev205-1.20.0
51
+
52
+ ## 0.1.2 - 2015-04-01
53
+
54
+ * [new feature] Changed bulk-load method from "via GCS" to direct-insert
55
+ * [new feature] added dynamic table creationg option
data/README.md CHANGED
@@ -28,7 +28,7 @@ OAuth flow for installed applications.
28
28
 
29
29
  | name | type | required? | default | description |
30
30
  |:--------------------------|:------------|:-----------|:-------------|:-----------------------|
31
- | mode | string | optional | append | [See below](#mode) |
31
+ | mode | string | optional | "append" | [See below](#mode) |
32
32
  | auth_method | string | optional | "private_key" | `private_key` , `json_key` or `compute_engine`
33
33
  | service_account_email | string | required when auth_method is private_key | | Your Google service account email
34
34
  | p12_keyfile | string | required when auth_method is private_key | | Fullpath of private key in P12(PKCS12) format |
@@ -65,6 +65,7 @@ Following options are same as [bq command-line tools](https://cloud.google.com/b
65
65
  ```yaml
66
66
  out:
67
67
  type: bigquery
68
+ mode: append
68
69
  auth_method: private_key # default
69
70
  service_account_email: ABCXYZ123ABCXYZ123.gserviceaccount.com
70
71
  p12_keyfile: /path/to/p12_keyfile.p12
@@ -85,18 +86,18 @@ out:
85
86
 
86
87
  4 modes are provided.
87
88
 
88
- #### append
89
+ ##### append
89
90
 
90
91
  default. When append mode, plugin will insert data into existing table.
91
92
 
92
- #### replace
93
+ ##### replace
93
94
 
94
95
  1. Load to temporary table.
95
96
  2. Copy temporary table to destination table. (WRITE_TRUNCATE)
96
97
 
97
98
  ```is_skip_job_result_check``` must be false when replace mode
98
99
 
99
- #### replace_backup
100
+ ##### replace_backup
100
101
 
101
102
  1. Load to temporary table.
102
103
  2. Copy destination table to backup table. (table_name_old)
@@ -104,7 +105,7 @@ default. When append mode, plugin will insert data into existing table.
104
105
 
105
106
  ```is_skip_job_result_check``` must be false when replace_backup mode.
106
107
 
107
- #### delete_in_advance
108
+ ##### delete_in_advance
108
109
 
109
110
  1. Delete destination table, if it exists.
110
111
  2. Load to destination table.
data/build.gradle CHANGED
@@ -2,6 +2,7 @@ plugins {
2
2
  id "com.jfrog.bintray" version "1.1"
3
3
  id "com.github.jruby-gradle.base" version "0.1.5"
4
4
  id "java"
5
+ id "checkstyle"
5
6
  }
6
7
  import com.github.jrubygradle.JRubyExec
7
8
  repositories {
@@ -15,11 +16,11 @@ configurations {
15
16
  sourceCompatibility = 1.7
16
17
  targetCompatibility = 1.7
17
18
 
18
- version = "0.2.0"
19
+ version = "0.2.1"
19
20
 
20
21
  dependencies {
21
- compile "org.embulk:embulk-core:0.7.1"
22
- provided "org.embulk:embulk-core:0.7.1"
22
+ compile "org.embulk:embulk-core:0.8.1"
23
+ provided "org.embulk:embulk-core:0.8.1"
23
24
 
24
25
  compile "com.google.http-client:google-http-client-jackson2:1.20.0"
25
26
  compile "com.google.apis:google-api-services-bigquery:v2-rev205-1.20.0"
@@ -34,6 +35,23 @@ task classpath(type: Copy, dependsOn: ["jar"]) {
34
35
  }
35
36
  clean { delete 'classpath' }
36
37
 
38
+ checkstyle {
39
+ configFile = file("${project.rootDir}/config/checkstyle/checkstyle.xml")
40
+ toolVersion = '6.14.1'
41
+ }
42
+ checkstyleMain {
43
+ configFile = file("${project.rootDir}/config/checkstyle/default.xml")
44
+ ignoreFailures = true
45
+ }
46
+ checkstyleTest {
47
+ configFile = file("${project.rootDir}/config/checkstyle/default.xml")
48
+ ignoreFailures = true
49
+ }
50
+ task checkstyle(type: Checkstyle) {
51
+ classpath = sourceSets.main.output + sourceSets.test.output
52
+ source = sourceSets.main.allJava + sourceSets.test.allJava
53
+ }
54
+
37
55
  task gem(type: JRubyExec, dependsOn: ["build", "gemspec", "classpath"]) {
38
56
  jrubyArgs "-rrubygems/gem_runner", "-eGem::GemRunner.new.run(ARGV)", "build"
39
57
  script "build/gemspec"
data/config/checkstyle/checkstyle.xml ADDED
@@ -0,0 +1,128 @@
1
+ <?xml version="1.0" encoding="UTF-8"?>
2
+ <!DOCTYPE module PUBLIC
3
+ "-//Puppy Crawl//DTD Check Configuration 1.3//EN"
4
+ "http://www.puppycrawl.com/dtds/configuration_1_3.dtd">
5
+ <module name="Checker">
6
+ <!-- https://github.com/facebook/presto/blob/master/src/checkstyle/checks.xml -->
7
+ <module name="FileTabCharacter"/>
8
+ <module name="NewlineAtEndOfFile">
9
+ <property name="lineSeparator" value="lf"/>
10
+ </module>
11
+ <module name="RegexpMultiline">
12
+ <property name="format" value="\r"/>
13
+ <property name="message" value="Line contains carriage return"/>
14
+ </module>
15
+ <module name="RegexpMultiline">
16
+ <property name="format" value=" \n"/>
17
+ <property name="message" value="Line has trailing whitespace"/>
18
+ </module>
19
+ <module name="RegexpMultiline">
20
+ <property name="format" value="\{\n\n"/>
21
+ <property name="message" value="Blank line after opening brace"/>
22
+ </module>
23
+ <module name="RegexpMultiline">
24
+ <property name="format" value="\n\n\s*\}"/>
25
+ <property name="message" value="Blank line before closing brace"/>
26
+ </module>
27
+ <module name="RegexpMultiline">
28
+ <property name="format" value="\n\n\n"/>
29
+ <property name="message" value="Multiple consecutive blank lines"/>
30
+ </module>
31
+ <module name="RegexpMultiline">
32
+ <property name="format" value="\n\n\Z"/>
33
+ <property name="message" value="Blank line before end of file"/>
34
+ </module>
35
+ <module name="RegexpMultiline">
36
+ <property name="format" value="Preconditions\.checkNotNull"/>
37
+ <property name="message" value="Use of checkNotNull"/>
38
+ </module>
39
+
40
+ <module name="TreeWalker">
41
+ <module name="EmptyBlock">
42
+ <property name="option" value="text"/>
43
+ <property name="tokens" value="
44
+ LITERAL_DO, LITERAL_ELSE, LITERAL_FINALLY, LITERAL_IF,
45
+ LITERAL_FOR, LITERAL_TRY, LITERAL_WHILE, INSTANCE_INIT, STATIC_INIT"/>
46
+ </module>
47
+ <module name="EmptyStatement"/>
48
+ <module name="EmptyForInitializerPad"/>
49
+ <module name="EmptyForIteratorPad">
50
+ <property name="option" value="space"/>
51
+ </module>
52
+ <module name="MethodParamPad">
53
+ <property name="allowLineBreaks" value="true"/>
54
+ <property name="option" value="nospace"/>
55
+ </module>
56
+ <module name="ParenPad"/>
57
+ <module name="TypecastParenPad"/>
58
+ <module name="NeedBraces"/>
59
+ <module name="LeftCurly">
60
+ <property name="option" value="nl"/>
61
+ <property name="tokens" value="CLASS_DEF, CTOR_DEF, INTERFACE_DEF, METHOD_DEF"/>
62
+ </module>
63
+ <module name="LeftCurly">
64
+ <property name="option" value="eol"/>
65
+ <property name="tokens" value="
66
+ LITERAL_CATCH, LITERAL_DO, LITERAL_ELSE, LITERAL_FINALLY, LITERAL_FOR,
67
+ LITERAL_IF, LITERAL_SWITCH, LITERAL_SYNCHRONIZED, LITERAL_TRY, LITERAL_WHILE"/>
68
+ </module>
69
+ <module name="RightCurly">
70
+ <property name="option" value="alone"/>
71
+ </module>
72
+ <module name="GenericWhitespace"/>
73
+ <module name="WhitespaceAfter"/>
74
+ <module name="NoWhitespaceBefore"/>
75
+
76
+ <module name="UpperEll"/>
77
+ <module name="DefaultComesLast"/>
78
+ <module name="ArrayTypeStyle"/>
79
+ <module name="MultipleVariableDeclarations"/>
80
+ <module name="ModifierOrder"/>
81
+ <module name="OneStatementPerLine"/>
82
+ <module name="StringLiteralEquality"/>
83
+ <module name="MutableException"/>
84
+ <module name="EqualsHashCode"/>
85
+ <module name="InnerAssignment"/>
86
+ <module name="InterfaceIsType"/>
87
+ <module name="HideUtilityClassConstructor"/>
88
+
89
+ <module name="MemberName"/>
90
+ <module name="LocalVariableName"/>
91
+ <module name="LocalFinalVariableName"/>
92
+ <module name="TypeName"/>
93
+ <module name="PackageName"/>
94
+ <module name="ParameterName"/>
95
+ <module name="StaticVariableName"/>
96
+ <module name="ClassTypeParameterName">
97
+ <property name="format" value="^[A-Z][0-9]?$"/>
98
+ </module>
99
+ <module name="MethodTypeParameterName">
100
+ <property name="format" value="^[A-Z][0-9]?$"/>
101
+ </module>
102
+
103
+ <module name="AvoidStarImport"/>
104
+ <module name="RedundantImport"/>
105
+ <module name="UnusedImports"/>
106
+ <module name="ImportOrder">
107
+ <property name="groups" value="*,javax,java"/>
108
+ <property name="separated" value="true"/>
109
+ <property name="option" value="bottom"/>
110
+ <property name="sortStaticImportsAlphabetically" value="true"/>
111
+ </module>
112
+
113
+ <module name="WhitespaceAround">
114
+ <property name="allowEmptyConstructors" value="true"/>
115
+ <property name="allowEmptyMethods" value="true"/>
116
+ <property name="ignoreEnhancedForColon" value="false"/>
117
+ <property name="tokens" value="
118
+ ASSIGN, BAND, BAND_ASSIGN, BOR, BOR_ASSIGN, BSR, BSR_ASSIGN,
119
+ BXOR, BXOR_ASSIGN, COLON, DIV, DIV_ASSIGN, EQUAL, GE, GT, LAND, LE,
120
+ LITERAL_ASSERT, LITERAL_CATCH, LITERAL_DO, LITERAL_ELSE,
121
+ LITERAL_FINALLY, LITERAL_FOR, LITERAL_IF, LITERAL_RETURN,
122
+ LITERAL_SYNCHRONIZED, LITERAL_TRY, LITERAL_WHILE,
123
+ LOR, LT, MINUS, MINUS_ASSIGN, MOD, MOD_ASSIGN, NOT_EQUAL,
124
+ PLUS, PLUS_ASSIGN, QUESTION, SL, SLIST, SL_ASSIGN, SR, SR_ASSIGN,
125
+ STAR, STAR_ASSIGN, TYPE_EXTENSION_AND"/>
126
+ </module>
127
+ </module>
128
+ </module>
data/config/checkstyle/default.xml ADDED
@@ -0,0 +1,108 @@
1
+ <?xml version="1.0" encoding="UTF-8"?>
2
+ <!DOCTYPE module PUBLIC
3
+ "-//Puppy Crawl//DTD Check Configuration 1.3//EN"
4
+ "http://www.puppycrawl.com/dtds/configuration_1_3.dtd">
5
+ <!--
6
+ This is a subset of ./checkstyle.xml which allows some loose styles
7
+ -->
8
+ <module name="Checker">
9
+ <module name="FileTabCharacter"/>
10
+ <module name="NewlineAtEndOfFile">
11
+ <property name="lineSeparator" value="lf"/>
12
+ </module>
13
+ <module name="RegexpMultiline">
14
+ <property name="format" value="\r"/>
15
+ <property name="message" value="Line contains carriage return"/>
16
+ </module>
17
+ <module name="RegexpMultiline">
18
+ <property name="format" value=" \n"/>
19
+ <property name="message" value="Line has trailing whitespace"/>
20
+ </module>
21
+ <module name="RegexpMultiline">
22
+ <property name="format" value="\n\n\n"/>
23
+ <property name="message" value="Multiple consecutive blank lines"/>
24
+ </module>
25
+ <module name="RegexpMultiline">
26
+ <property name="format" value="\n\n\Z"/>
27
+ <property name="message" value="Blank line before end of file"/>
28
+ </module>
29
+
30
+ <module name="TreeWalker">
31
+ <module name="EmptyBlock">
32
+ <property name="option" value="text"/>
33
+ <property name="tokens" value="
34
+ LITERAL_DO, LITERAL_ELSE, LITERAL_FINALLY, LITERAL_IF,
35
+ LITERAL_FOR, LITERAL_TRY, LITERAL_WHILE, INSTANCE_INIT, STATIC_INIT"/>
36
+ </module>
37
+ <module name="EmptyStatement"/>
38
+ <module name="EmptyForInitializerPad"/>
39
+ <module name="EmptyForIteratorPad">
40
+ <property name="option" value="space"/>
41
+ </module>
42
+ <module name="MethodParamPad">
43
+ <property name="allowLineBreaks" value="true"/>
44
+ <property name="option" value="nospace"/>
45
+ </module>
46
+ <module name="ParenPad"/>
47
+ <module name="TypecastParenPad"/>
48
+ <module name="NeedBraces"/>
49
+ <module name="LeftCurly">
50
+ <property name="option" value="nl"/>
51
+ <property name="tokens" value="CLASS_DEF, CTOR_DEF, INTERFACE_DEF, METHOD_DEF"/>
52
+ </module>
53
+ <module name="LeftCurly">
54
+ <property name="option" value="eol"/>
55
+ <property name="tokens" value="
56
+ LITERAL_CATCH, LITERAL_DO, LITERAL_ELSE, LITERAL_FINALLY, LITERAL_FOR,
57
+ LITERAL_IF, LITERAL_SWITCH, LITERAL_SYNCHRONIZED, LITERAL_TRY, LITERAL_WHILE"/>
58
+ </module>
59
+ <module name="RightCurly">
60
+ <property name="option" value="alone"/>
61
+ </module>
62
+ <module name="GenericWhitespace"/>
63
+ <module name="WhitespaceAfter"/>
64
+ <module name="NoWhitespaceBefore"/>
65
+
66
+ <module name="UpperEll"/>
67
+ <module name="DefaultComesLast"/>
68
+ <module name="ArrayTypeStyle"/>
69
+ <module name="MultipleVariableDeclarations"/>
70
+ <module name="ModifierOrder"/>
71
+ <module name="OneStatementPerLine"/>
72
+ <module name="StringLiteralEquality"/>
73
+ <module name="MutableException"/>
74
+ <module name="EqualsHashCode"/>
75
+ <module name="InnerAssignment"/>
76
+ <module name="InterfaceIsType"/>
77
+ <module name="HideUtilityClassConstructor"/>
78
+
79
+ <module name="MemberName"/>
80
+ <module name="LocalVariableName"/>
81
+ <module name="LocalFinalVariableName"/>
82
+ <module name="TypeName"/>
83
+ <module name="PackageName"/>
84
+ <module name="ParameterName"/>
85
+ <module name="StaticVariableName"/>
86
+ <module name="ClassTypeParameterName">
87
+ <property name="format" value="^[A-Z][0-9]?$"/>
88
+ </module>
89
+ <module name="MethodTypeParameterName">
90
+ <property name="format" value="^[A-Z][0-9]?$"/>
91
+ </module>
92
+
93
+ <module name="WhitespaceAround">
94
+ <property name="allowEmptyConstructors" value="true"/>
95
+ <property name="allowEmptyMethods" value="true"/>
96
+ <property name="ignoreEnhancedForColon" value="false"/>
97
+ <property name="tokens" value="
98
+ ASSIGN, BAND, BAND_ASSIGN, BOR, BOR_ASSIGN, BSR, BSR_ASSIGN,
99
+ BXOR, BXOR_ASSIGN, COLON, DIV, DIV_ASSIGN, EQUAL, GE, GT, LAND, LE,
100
+ LITERAL_ASSERT, LITERAL_CATCH, LITERAL_DO, LITERAL_ELSE,
101
+ LITERAL_FINALLY, LITERAL_FOR, LITERAL_IF, LITERAL_RETURN,
102
+ LITERAL_SYNCHRONIZED, LITERAL_TRY, LITERAL_WHILE,
103
+ LOR, LT, MINUS, MINUS_ASSIGN, MOD, MOD_ASSIGN, NOT_EQUAL,
104
+ PLUS, PLUS_ASSIGN, QUESTION, SL, SLIST, SL_ASSIGN, SR, SR_ASSIGN,
105
+ STAR, STAR_ASSIGN, TYPE_EXTENSION_AND"/>
106
+ </module>
107
+ </module>
108
+ </module>
data/gradle/wrapper/gradle-wrapper.jar CHANGED
Binary file
data/gradle/wrapper/gradle-wrapper.properties CHANGED
@@ -1,6 +1,6 @@
1
- #Tue Aug 11 00:26:20 PDT 2015
1
+ #Wed Jan 13 12:41:02 JST 2016
2
2
  distributionBase=GRADLE_USER_HOME
3
3
  distributionPath=wrapper/dists
4
4
  zipStoreBase=GRADLE_USER_HOME
5
5
  zipStorePath=wrapper/dists
6
- distributionUrl=https\://services.gradle.org/distributions/gradle-2.6-bin.zip
6
+ distributionUrl=https\://services.gradle.org/distributions/gradle-2.10-bin.zip
data/src/main/java/org/embulk/output/BigqueryAuthentication.java CHANGED
@@ -47,9 +47,11 @@ public class BigqueryAuthentication
47
47
 
48
48
  if (authMethod.toLowerCase().equals("compute_engine")) {
49
49
  this.credentials = getComputeCredential();
50
- } else if(authMethod.toLowerCase().equals("json_key")) {
50
+ }
51
+ else if (authMethod.toLowerCase().equals("json_key")) {
51
52
  this.credentials = getServiceAccountCredentialFromJsonFile();
52
- } else {
53
+ }
54
+ else {
53
55
  this.credentials = getServiceAccountCredential();
54
56
  }
55
57
  }
@@ -111,4 +113,4 @@ public class BigqueryAuthentication
111
113
 
112
114
  return client;
113
115
  }
114
- }
116
+ }
data/src/main/java/org/embulk/output/BigqueryOutputPlugin.java CHANGED
@@ -146,7 +146,7 @@ public class BigqueryOutputPlugin
146
146
  }
147
147
 
148
148
  private final Logger log = Exec.getLogger(BigqueryOutputPlugin.class);
149
- private final static String temporaryTableSuffix = Long.toString(System.currentTimeMillis());
149
+ private static final String temporaryTableSuffix = Long.toString(System.currentTimeMillis());
150
150
  private static BigqueryWriter bigQueryWriter;
151
151
 
152
152
  @Override
@@ -161,7 +161,8 @@ public class BigqueryOutputPlugin
161
161
  }
162
162
  try {
163
163
  task.setP12Keyfile(Optional.of(LocalFile.of(task.getP12KeyfilePath().get())));
164
- } catch (IOException ex) {
164
+ }
165
+ catch (IOException ex) {
165
166
  throw Throwables.propagate(ex);
166
167
  }
167
168
  }
@@ -172,7 +173,8 @@ public class BigqueryOutputPlugin
172
173
  }
173
174
  try {
174
175
  task.setSchemaFile(Optional.of(LocalFile.of(task.getSchemaPath().get())));
175
- } catch (IOException ex) {
176
+ }
177
+ catch (IOException ex) {
176
178
  throw Throwables.propagate(ex);
177
179
  }
178
180
  }
@@ -181,7 +183,8 @@ public class BigqueryOutputPlugin
181
183
  if (!task.getJsonKeyfile().isPresent()) {
182
184
  throw new ConfigException("If auth_method is json_key, you have to set json_keyfile");
183
185
  }
184
- } else if (task.getAuthMethod().getString().equals("private_key")) {
186
+ }
187
+ else if (task.getAuthMethod().getString().equals("private_key")) {
185
188
  if (!task.getP12Keyfile().isPresent() || !task.getServiceAccountEmail().isPresent()) {
186
189
  throw new ConfigException("If auth_method is private_key, you have to set both service_account_email and p12_keyfile");
187
190
  }
@@ -200,7 +203,7 @@ public class BigqueryOutputPlugin
200
203
  }
201
204
 
202
205
  try {
203
- bigQueryWriter = new BigqueryWriter.Builder (
206
+ bigQueryWriter = new BigqueryWriter.Builder(
204
207
  task.getAuthMethod().getString(),
205
208
  task.getServiceAccountEmail(),
206
209
  task.getP12Keyfile().transform(localFileToPathString()),
@@ -222,7 +225,8 @@ public class BigqueryOutputPlugin
222
225
 
223
226
  bigQueryWriter.checkConfig(task.getProject(), task.getDataset(), task.getTable());
224
227
 
225
- } catch (IOException | GeneralSecurityException ex) {
228
+ }
229
+ catch (IOException | GeneralSecurityException ex) {
226
230
  throw new ConfigException(ex);
227
231
  }
228
232
  // non-retryable (non-idempotent) output:
@@ -242,7 +246,8 @@ public class BigqueryOutputPlugin
242
246
  if (mode == Mode.delete_in_advance) {
243
247
  try {
244
248
  bigQueryWriter.deleteTable(project, dataset, generateTableName(tableName));
245
- } catch (IOException ex) {
249
+ }
250
+ catch (IOException ex) {
246
251
  log.warn(ex.getMessage());
247
252
  }
248
253
  }
@@ -255,13 +260,16 @@ public class BigqueryOutputPlugin
255
260
  bigQueryWriter.replaceTable(project, dataset, generateTableName(tableName) + "_old", generateTableName(tableName));
256
261
  }
257
262
  bigQueryWriter.replaceTable(project, dataset, generateTableName(tableName), generateTemporaryTableName(tableName));
258
- } catch (TimeoutException | BigqueryWriter.JobFailedException | IOException ex) {
263
+ }
264
+ catch (TimeoutException | BigqueryWriter.JobFailedException | IOException ex) {
259
265
  log.error(ex.getMessage());
260
266
  throw Throwables.propagate(ex);
261
- } finally {
267
+ }
268
+ finally {
262
269
  try {
263
270
  bigQueryWriter.deleteTable(project, dataset, generateTemporaryTableName(tableName));
264
- } catch (IOException ex) {
271
+ }
272
+ catch (IOException ex) {
265
273
  log.warn(ex.getMessage());
266
274
  }
267
275
  }
@@ -328,7 +336,8 @@ public class BigqueryOutputPlugin
328
336
  }
329
337
  log.info(String.format("Writing file [%s]", filePath));
330
338
  output = new BufferedOutputStream(new FileOutputStream(filePath));
331
- } catch (FileNotFoundException ex) {
339
+ }
340
+ catch (FileNotFoundException ex) {
332
341
  throw Throwables.propagate(ex);
333
342
  }
334
343
  fileIndex++;
@@ -339,7 +348,8 @@ public class BigqueryOutputPlugin
339
348
  if (output != null) {
340
349
  try {
341
350
  output.close();
342
- } catch (IOException ex) {
351
+ }
352
+ catch (IOException ex) {
343
353
  throw Throwables.propagate(ex);
344
354
  }
345
355
  }
@@ -349,9 +359,11 @@ public class BigqueryOutputPlugin
349
359
  {
350
360
  try {
351
361
  output.write(buffer.array(), buffer.offset(), buffer.limit());
352
- } catch (IOException ex) {
362
+ }
363
+ catch (IOException ex) {
353
364
  throw Throwables.propagate(ex);
354
- } finally {
365
+ }
366
+ finally {
355
367
  buffer.release();
356
368
  }
357
369
  }
@@ -367,7 +379,8 @@ public class BigqueryOutputPlugin
367
379
  log.info(String.format("Delete local file [%s]", filePath));
368
380
  file.delete();
369
381
  }
370
- } catch (NoSuchAlgorithmException | TimeoutException | BigqueryWriter.JobFailedException | IOException ex) {
382
+ }
383
+ catch (NoSuchAlgorithmException | TimeoutException | BigqueryWriter.JobFailedException | IOException ex) {
371
384
  log.error(ex.getMessage());
372
385
  throw Throwables.propagate(ex);
373
386
  }
@@ -447,15 +460,24 @@ public class BigqueryOutputPlugin
447
460
  append("append"),
448
461
  delete_in_advance("delete_in_advance") {
449
462
  @Override
450
- public boolean isDeleteInAdvance() { return true; }
463
+ public boolean isDeleteInAdvance()
464
+ {
465
+ return true;
466
+ }
451
467
  },
452
468
  replace("replace") {
453
469
  @Override
454
- public boolean isReplaceMode() { return true; }
470
+ public boolean isReplaceMode()
471
+ {
472
+ return true;
473
+ }
455
474
  },
456
475
  replace_backup("replace_backup") {
457
476
  @Override
458
- public boolean isReplaceMode() { return true; }
477
+ public boolean isReplaceMode()
478
+ {
479
+ return true;
480
+ }
459
481
  };
460
482
 
461
483
  private final String string;
@@ -465,8 +487,17 @@ public class BigqueryOutputPlugin
465
487
  this.string = string;
466
488
  }
467
489
 
468
- public String getString() { return string; }
469
- public boolean isReplaceMode() { return false; }
470
- public boolean isDeleteInAdvance() { return true; }
490
+ public String getString()
491
+ {
492
+ return string;
493
+ }
494
+ public boolean isReplaceMode()
495
+ {
496
+ return false;
497
+ }
498
+ public boolean isDeleteInAdvance()
499
+ {
500
+ return true;
501
+ }
471
502
  }
472
503
  }
data/src/main/java/org/embulk/output/BigqueryWriter.java CHANGED
@@ -72,7 +72,8 @@ public class BigqueryWriter
72
72
 
73
73
  if (autoCreateTable) {
74
74
  this.tableSchema = createTableSchema();
75
- } else {
75
+ }
76
+ else {
76
77
  this.tableSchema = null;
77
78
  }
78
79
  }
@@ -100,7 +101,8 @@ public class BigqueryWriter
100
101
  log.info(String.format("Job statistics [%s]", statistics.getLoad()));
101
102
  }
102
103
  return jobStatus;
103
- } catch (IOException ex) {
104
+ }
105
+ catch (IOException ex) {
104
106
  log.warn(ex.getMessage());
105
107
  return "UNKNOWN";
106
108
  }
@@ -118,14 +120,17 @@ public class BigqueryWriter
118
120
  if (jobStatus.equals("DONE")) {
119
121
  log.info(String.format("Job completed successfully. job id:[%s] elapsed_time:%dms status:[%s]", jobRef.getJobId(), elapsedTime, "SUCCESS"));
120
122
  break;
121
- } else if (elapsedTime > jobStatusMaxPollingTime * 1000) {
123
+ }
124
+ else if (elapsedTime > jobStatusMaxPollingTime * 1000) {
122
125
  throw new TimeoutException(String.format("Checking job status...Timeout. job id:[%s] elapsed_time:%dms status:[%s]", jobRef.getJobId(), elapsedTime, "TIMEOUT"));
123
- } else {
126
+ }
127
+ else {
124
128
  log.info(String.format("Checking job status... job id:[%s] elapsed_time:%dms status:[%s]", jobRef.getJobId(), elapsedTime, jobStatus));
125
129
  }
126
130
  Thread.sleep(jobStatusPollingInterval * 1000);
127
131
  }
128
- } catch (InterruptedException ex) {
132
+ }
133
+ catch (InterruptedException ex) {
129
134
  log.warn(ex.getMessage());
130
135
  }
131
136
  }
@@ -171,13 +176,15 @@ public class BigqueryWriter
171
176
 
172
177
  try {
173
178
  jobRef = insert.execute().getJobReference();
174
- } catch (IllegalStateException ex) {
179
+ }
180
+ catch (IllegalStateException ex) {
175
181
  throw new JobFailedException(ex.getMessage());
176
182
  }
177
183
  log.info(String.format("Job executed. job id:[%s] file:[%s]", jobRef.getJobId(), localFilePath));
178
184
  if (isSkipJobResultCheck) {
179
185
  log.info(String.format("Skip job status check. job id:[%s]", jobRef.getJobId()));
180
- } else {
186
+ }
187
+ else {
181
188
  getJobStatusUntilDone(project, jobRef);
182
189
  }
183
190
  }
@@ -203,19 +210,22 @@ public class BigqueryWriter
203
210
 
204
211
  try {
205
212
  jobRef = insert.execute().getJobReference();
206
- } catch (IllegalStateException ex) {
213
+ }
214
+ catch (IllegalStateException ex) {
207
215
  throw new JobFailedException(ex.getMessage());
208
216
  }
209
217
  log.info(String.format("Job executed. job id:[%s]", jobRef.getJobId()));
210
218
  getJobStatusUntilDone(project, jobRef);
211
219
  }
212
220
 
213
- public void deleteTable(String project, String dataset, String table) throws IOException {
221
+ public void deleteTable(String project, String dataset, String table) throws IOException
222
+ {
214
223
  try {
215
224
  Tables.Delete delete = bigQueryClient.tables().delete(project, dataset, table);
216
225
  delete.execute();
217
226
  log.info(String.format("Table deleted. project:%s dataset:%s table:%s", delete.getProjectId(), delete.getDatasetId(), delete.getTableId()));
218
- } catch (GoogleJsonResponseException ex) {
227
+ }
228
+ catch (GoogleJsonResponseException ex) {
219
229
  log.warn(ex.getMessage());
220
230
  }
221
231
  }
@@ -238,7 +248,8 @@ public class BigqueryWriter
238
248
  config.setSchema(tableSchema);
239
249
  config.setCreateDisposition("CREATE_IF_NEEDED");
240
250
  log.info(String.format("table:[%s] will be create if not exists", table));
241
- } else {
251
+ }
252
+ else {
242
253
  config.setCreateDisposition("CREATE_NEVER");
243
254
  }
244
255
  return config;
@@ -252,7 +263,8 @@ public class BigqueryWriter
252
263
 
253
264
  if (append) {
254
265
  config.setWriteDisposition("WRITE_APPEND");
255
- } else {
266
+ }
267
+ else {
256
268
  config.setWriteDisposition("WRITE_TRUNCATE");
257
269
  }
258
270
 
@@ -294,7 +306,8 @@ public class BigqueryWriter
294
306
  ObjectMapper mapper = new ObjectMapper();
295
307
  List<TableFieldSchema> fields = mapper.readValue(stream, new TypeReference<List<TableFieldSchema>>() {});
296
308
  return new TableSchema().setFields(fields);
297
- } finally {
309
+ }
310
+ finally {
298
311
  if (stream != null) {
299
312
  stream.close();
300
313
  }
@@ -306,7 +319,8 @@ public class BigqueryWriter
306
319
  Tables tableRequest = bigQueryClient.tables();
307
320
  try {
308
321
  Table tableData = tableRequest.get(project, dataset, table).execute();
309
- } catch (GoogleJsonResponseException ex) {
322
+ }
323
+ catch (GoogleJsonResponseException ex) {
310
324
  return false;
311
325
  }
312
326
  return true;
@@ -317,13 +331,15 @@ public class BigqueryWriter
317
331
  if (autoCreateTable) {
318
332
  if (!schemaPath.isPresent()) {
319
333
  throw new FileNotFoundException("schema_file is empty");
320
- } else {
334
+ }
335
+ else {
321
336
  File file = new File(schemaPath.orNull());
322
337
  if (!file.exists()) {
323
338
  throw new FileNotFoundException("Can not load schema file.");
324
339
  }
325
340
  }
326
- } else {
341
+ }
342
+ else {
327
343
  if (!isExistTable(project, dataset, table)) {
328
344
  throw new IOException(String.format("table [%s] is not exists", table));
329
345
  }
@@ -347,7 +363,8 @@ public class BigqueryWriter
347
363
 
348
364
  byte[] encoded = (hashedBytes);
349
365
  return new String(encoded);
350
- } finally {
366
+ }
367
+ finally {
351
368
  stream.close();
352
369
  }
353
370
  }
@@ -490,7 +507,8 @@ public class BigqueryWriter
490
507
 
491
508
  public class JobFailedException extends RuntimeException
492
509
  {
493
- public JobFailedException(String message) {
510
+ public JobFailedException(String message)
511
+ {
494
512
  super(message);
495
513
  }
496
514
  }
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: embulk-output-bigquery
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.0
4
+ version: 0.2.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Satoshi Akama
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2016-01-26 00:00:00.000000000 Z
11
+ date: 2016-01-28 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  requirement: !ruby/object:Gem::Requirement
@@ -46,9 +46,11 @@ extensions: []
46
46
  extra_rdoc_files: []
47
47
  files:
48
48
  - .gitignore
49
- - ChangeLog
49
+ - CHANGELOG.md
50
50
  - README.md
51
51
  - build.gradle
52
+ - config/checkstyle/checkstyle.xml
53
+ - config/checkstyle/default.xml
52
54
  - gradle/wrapper/gradle-wrapper.jar
53
55
  - gradle/wrapper/gradle-wrapper.properties
54
56
  - gradlew
@@ -63,7 +65,7 @@ files:
63
65
  - src/test/java/org/embulk/output/TestBigqueryWriter.java
64
66
  - classpath/commons-codec-1.3.jar
65
67
  - classpath/commons-logging-1.1.1.jar
66
- - classpath/embulk-output-bigquery-0.2.0.jar
68
+ - classpath/embulk-output-bigquery-0.2.1.jar
67
69
  - classpath/google-api-client-1.20.0.jar
68
70
  - classpath/google-api-services-bigquery-v2-rev205-1.20.0.jar
69
71
  - classpath/google-http-client-1.20.0.jar
data/ChangeLog DELETED
@@ -1,50 +0,0 @@
1
- Release 0.2.0 - 2016-01-26
2
-
3
- * Added mode parameters and support 4 modes(append, replace, replace_backup, delete_in_advance). @joker1007 thanks!
4
-
5
- Release 0.1.11 - 2015-11-16
6
-
7
- * Change error result display for easy investigation.
8
-
9
- Release 0.1.10 - 2015-10-06
10
-
11
- * Added new auth method - json_keyfile of GCP(Google Cloud Platform)'s service account
12
-
13
- Release 0.1.9 - 2015-08-19
14
-
15
- * Upgraded embulk version to 0.7.1
16
-
17
- Release 0.1.8 - 2015-08-19
18
-
19
- * Supported mapreduce-executor. @frsyuki thanks!
20
- * Fixed job_id generation logic
21
- * Refactored
22
-
23
- Release 0.1.7 - 2015-05-20
24
-
25
- * Added allow_quoted_newlines option
26
- * Upgraded embulk version to 0.6.8
27
-
28
- Release 0.1.6 - 2015-04-23
29
-
30
- * Added ignore_unknown_values option to job_id generation logic.
31
-
32
- Release 0.1.5 - 2015-04-23
33
-
34
- * Added ignore_unknown_values option. @takus thanks!
35
-
36
- Release 0.1.4 - 2015-04-21
37
-
38
- * Added prevent_duplicate_insert option
39
-
40
- Release 0.1.3 - 2015-04-06
41
-
42
- * Added new auth method - pre-defined access token of GCE(Google Compute Engine)
43
- * Updated Google provided libraries
44
- * http-client:google-http-client-jackson2 from 1.19.0 to 1.20.0
45
- * apis:google-api-services-bigquery from v2-rev193-1.19.1 to v2-rev205-1.20.0
46
-
47
- Release 0.1.2 - 2015-04-01
48
-
49
- * Changed bulk-load method from "via GCS" to direct-insert
50
- * added dynamic table creationg option