embulk-output-bigquery 0.2.0 → 0.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (13) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +55 -0
  3. data/README.md +6 -5
  4. data/build.gradle +21 -3
  5. data/config/checkstyle/checkstyle.xml +128 -0
  6. data/config/checkstyle/default.xml +108 -0
  7. data/gradle/wrapper/gradle-wrapper.jar +0 -0
  8. data/gradle/wrapper/gradle-wrapper.properties +2 -2
  9. data/src/main/java/org/embulk/output/BigqueryAuthentication.java +5 -3
  10. data/src/main/java/org/embulk/output/BigqueryOutputPlugin.java +52 -21
  11. data/src/main/java/org/embulk/output/BigqueryWriter.java +36 -18
  12. metadata +6 -4
  13. data/ChangeLog +0 -50
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 9e4b4a5467fb4589a3d88541af95ee5b002a5f9d
4
- data.tar.gz: 2845c3d881b88183bdf3355647fbfecd6e455372
3
+ metadata.gz: 7e6de19fdc976904bd2ac46aa529e030f9ecbe48
4
+ data.tar.gz: d279269a8914553203a1d2bc06832ed4201bedf3
5
5
  SHA512:
6
- metadata.gz: 61e5481dd18b750555f3462d680cb2b4d680ce768683780e4b2ce2cac6e66134c7eb38227adf8e48a4655447e27e166f14487c129fe9154a164131a4be06270b
7
- data.tar.gz: 09036cb1d4b20556cb36e11839c7807b00ae453976b4192f9c6f489284aa5f1cbe48b6c42e5b9149b1ed3c3d04a92ee9546cce030b3d9e0c0929f70e365d3ae7
6
+ metadata.gz: 3e7f86e4b963012e0a55227b53262c141209d5aaed5c06b82d1d3da844f467c2b0e53c750e31eec38963459e58e02ba54b82fbb39c14084df241428b5245a220
7
+ data.tar.gz: 60850b5ce34b7b626cf9766a537196d160ae1c1bd0365f119817f50977f7a171762e6562ac987cdc2655fc75892dd6beafceebd47305818e625330aaabb1911b
data/CHANGELOG.md ADDED
@@ -0,0 +1,55 @@
1
+ ## 0.2.1 - 2016-01-28
2
+
3
+ * [maintenance] Upgraded Embulk version to 0.8.1 [#22](https://github.com/embulk/embulk-output-bigquery/pull/22). @joker1007 thanks!
4
+ * [maintenance] Formatted code style by checkstyle [#23](https://github.com/embulk/embulk-output-bigquery/pull/23)
5
+
6
+ ## 0.2.0 - 2016-01-26
7
+
8
+ * [new feature] Added mode parameters and support 4 modes(append, replace, replace_backup, delete_in_advance). [#20](https://github.com/embulk/embulk-output-bigquery/pull/20) [#21](https://github.com/embulk/embulk-output-bigquery/pull/21) @joker1007 thanks!
9
+
10
+ ## 0.1.11 - 2015-11-16
11
+
12
+ * [maintenance] Change error result display for easy investigation. [#18](https://github.com/embulk/embulk-output-bigquery/pull/18)
13
+
14
+ ## 0.1.10 - 2015-10-06
15
+
16
+ * [new feature] Added new auth method - json_keyfile of GCP(Google Cloud Platform)'s service account [#17](https://github.com/embulk/embulk-output-bigquery/pull/17)
17
+
18
+ ## 0.1.9 - 2015-08-19
19
+
20
+ * [maintenance] Upgraded Embulk version to 0.7.1
21
+
22
+ ## 0.1.8 - 2015-08-19
23
+
24
+ * [new feature] Supported mapreduce-executor. @frsyuki thanks! [#13](https://github.com/embulk/embulk-output-bigquery/pull/13)
25
+ * [maintenance] Fixed job_id generation logic [#15](https://github.com/embulk/embulk-output-bigquery/pull/15)
26
+ * [maintenance] Refactored [#11](https://github.com/embulk/embulk-output-bigquery/pull/11)
27
+
28
+ ## 0.1.7 - 2015-05-20
29
+
30
+ * [new feature] Added allow_quoted_newlines option [#10](https://github.com/embulk/embulk-output-bigquery/pull/10)
31
+ * [maintenance] Upgraded embulk version to 0.6.8
32
+
33
+ ## 0.1.6 - 2015-04-23
34
+
35
+ * [new feature] Added ignore_unknown_values option to job_id generation logic. [#9](https://github.com/embulk/embulk-output-bigquery/pull/9)
36
+
37
+ ## 0.1.5 - 2015-04-23
38
+
39
+ * [new feature] Added ignore_unknown_values option. [#8](https://github.com/embulk/embulk-output-bigquery/pull/8) @takus thanks!
40
+
41
+ ## 0.1.4 - 2015-04-21
42
+
43
+ * [new feature] Added prevent_duplicate_insert option
44
+
45
+ ## 0.1.3 - 2015-04-06
46
+
47
+ * [new feature] Added new auth method - pre-defined access token of GCE(Google Compute Engine)
48
+ * [maintenance] Updated Google provided libraries
49
+ * http-client:google-http-client-jackson2 from 1.19.0 to 1.20.0
50
+ * apis:google-api-services-bigquery from v2-rev193-1.19.1 to v2-rev205-1.20.0
51
+
52
+ ## 0.1.2 - 2015-04-01
53
+
54
+ * [new feature] Changed bulk-load method from "via GCS" to direct-insert
55
+ * [new feature] added dynamic table creationg option
data/README.md CHANGED
@@ -28,7 +28,7 @@ OAuth flow for installed applications.
28
28
 
29
29
  | name | type | required? | default | description |
30
30
  |:--------------------------|:------------|:-----------|:-------------|:-----------------------|
31
- | mode | string | optional | append | [See below](#mode) |
31
+ | mode | string | optional | "append" | [See below](#mode) |
32
32
  | auth_method | string | optional | "private_key" | `private_key` , `json_key` or `compute_engine`
33
33
  | service_account_email | string | required when auth_method is private_key | | Your Google service account email
34
34
  | p12_keyfile | string | required when auth_method is private_key | | Fullpath of private key in P12(PKCS12) format |
@@ -65,6 +65,7 @@ Following options are same as [bq command-line tools](https://cloud.google.com/b
65
65
  ```yaml
66
66
  out:
67
67
  type: bigquery
68
+ mode: append
68
69
  auth_method: private_key # default
69
70
  service_account_email: ABCXYZ123ABCXYZ123.gserviceaccount.com
70
71
  p12_keyfile: /path/to/p12_keyfile.p12
@@ -85,18 +86,18 @@ out:
85
86
 
86
87
  4 modes are provided.
87
88
 
88
- #### append
89
+ ##### append
89
90
 
90
91
  default. When append mode, plugin will insert data into existing table.
91
92
 
92
- #### replace
93
+ ##### replace
93
94
 
94
95
  1. Load to temporary table.
95
96
  2. Copy temporary table to destination table. (WRITE_TRUNCATE)
96
97
 
97
98
  ```is_skip_job_result_check``` must be false when replace mode
98
99
 
99
- #### replace_backup
100
+ ##### replace_backup
100
101
 
101
102
  1. Load to temporary table.
102
103
  2. Copy destination table to backup table. (table_name_old)
@@ -104,7 +105,7 @@ default. When append mode, plugin will insert data into existing table.
104
105
 
105
106
  ```is_skip_job_result_check``` must be false when replace_backup mode.
106
107
 
107
- #### delete_in_advance
108
+ ##### delete_in_advance
108
109
 
109
110
  1. Delete destination table, if it exists.
110
111
  2. Load to destination table.
data/build.gradle CHANGED
@@ -2,6 +2,7 @@ plugins {
2
2
  id "com.jfrog.bintray" version "1.1"
3
3
  id "com.github.jruby-gradle.base" version "0.1.5"
4
4
  id "java"
5
+ id "checkstyle"
5
6
  }
6
7
  import com.github.jrubygradle.JRubyExec
7
8
  repositories {
@@ -15,11 +16,11 @@ configurations {
15
16
  sourceCompatibility = 1.7
16
17
  targetCompatibility = 1.7
17
18
 
18
- version = "0.2.0"
19
+ version = "0.2.1"
19
20
 
20
21
  dependencies {
21
- compile "org.embulk:embulk-core:0.7.1"
22
- provided "org.embulk:embulk-core:0.7.1"
22
+ compile "org.embulk:embulk-core:0.8.1"
23
+ provided "org.embulk:embulk-core:0.8.1"
23
24
 
24
25
  compile "com.google.http-client:google-http-client-jackson2:1.20.0"
25
26
  compile "com.google.apis:google-api-services-bigquery:v2-rev205-1.20.0"
@@ -34,6 +35,23 @@ task classpath(type: Copy, dependsOn: ["jar"]) {
34
35
  }
35
36
  clean { delete 'classpath' }
36
37
 
38
+ checkstyle {
39
+ configFile = file("${project.rootDir}/config/checkstyle/checkstyle.xml")
40
+ toolVersion = '6.14.1'
41
+ }
42
+ checkstyleMain {
43
+ configFile = file("${project.rootDir}/config/checkstyle/default.xml")
44
+ ignoreFailures = true
45
+ }
46
+ checkstyleTest {
47
+ configFile = file("${project.rootDir}/config/checkstyle/default.xml")
48
+ ignoreFailures = true
49
+ }
50
+ task checkstyle(type: Checkstyle) {
51
+ classpath = sourceSets.main.output + sourceSets.test.output
52
+ source = sourceSets.main.allJava + sourceSets.test.allJava
53
+ }
54
+
37
55
  task gem(type: JRubyExec, dependsOn: ["build", "gemspec", "classpath"]) {
38
56
  jrubyArgs "-rrubygems/gem_runner", "-eGem::GemRunner.new.run(ARGV)", "build"
39
57
  script "build/gemspec"
data/config/checkstyle/checkstyle.xml ADDED
@@ -0,0 +1,128 @@
1
+ <?xml version="1.0" encoding="UTF-8"?>
2
+ <!DOCTYPE module PUBLIC
3
+ "-//Puppy Crawl//DTD Check Configuration 1.3//EN"
4
+ "http://www.puppycrawl.com/dtds/configuration_1_3.dtd">
5
+ <module name="Checker">
6
+ <!-- https://github.com/facebook/presto/blob/master/src/checkstyle/checks.xml -->
7
+ <module name="FileTabCharacter"/>
8
+ <module name="NewlineAtEndOfFile">
9
+ <property name="lineSeparator" value="lf"/>
10
+ </module>
11
+ <module name="RegexpMultiline">
12
+ <property name="format" value="\r"/>
13
+ <property name="message" value="Line contains carriage return"/>
14
+ </module>
15
+ <module name="RegexpMultiline">
16
+ <property name="format" value=" \n"/>
17
+ <property name="message" value="Line has trailing whitespace"/>
18
+ </module>
19
+ <module name="RegexpMultiline">
20
+ <property name="format" value="\{\n\n"/>
21
+ <property name="message" value="Blank line after opening brace"/>
22
+ </module>
23
+ <module name="RegexpMultiline">
24
+ <property name="format" value="\n\n\s*\}"/>
25
+ <property name="message" value="Blank line before closing brace"/>
26
+ </module>
27
+ <module name="RegexpMultiline">
28
+ <property name="format" value="\n\n\n"/>
29
+ <property name="message" value="Multiple consecutive blank lines"/>
30
+ </module>
31
+ <module name="RegexpMultiline">
32
+ <property name="format" value="\n\n\Z"/>
33
+ <property name="message" value="Blank line before end of file"/>
34
+ </module>
35
+ <module name="RegexpMultiline">
36
+ <property name="format" value="Preconditions\.checkNotNull"/>
37
+ <property name="message" value="Use of checkNotNull"/>
38
+ </module>
39
+
40
+ <module name="TreeWalker">
41
+ <module name="EmptyBlock">
42
+ <property name="option" value="text"/>
43
+ <property name="tokens" value="
44
+ LITERAL_DO, LITERAL_ELSE, LITERAL_FINALLY, LITERAL_IF,
45
+ LITERAL_FOR, LITERAL_TRY, LITERAL_WHILE, INSTANCE_INIT, STATIC_INIT"/>
46
+ </module>
47
+ <module name="EmptyStatement"/>
48
+ <module name="EmptyForInitializerPad"/>
49
+ <module name="EmptyForIteratorPad">
50
+ <property name="option" value="space"/>
51
+ </module>
52
+ <module name="MethodParamPad">
53
+ <property name="allowLineBreaks" value="true"/>
54
+ <property name="option" value="nospace"/>
55
+ </module>
56
+ <module name="ParenPad"/>
57
+ <module name="TypecastParenPad"/>
58
+ <module name="NeedBraces"/>
59
+ <module name="LeftCurly">
60
+ <property name="option" value="nl"/>
61
+ <property name="tokens" value="CLASS_DEF, CTOR_DEF, INTERFACE_DEF, METHOD_DEF"/>
62
+ </module>
63
+ <module name="LeftCurly">
64
+ <property name="option" value="eol"/>
65
+ <property name="tokens" value="
66
+ LITERAL_CATCH, LITERAL_DO, LITERAL_ELSE, LITERAL_FINALLY, LITERAL_FOR,
67
+ LITERAL_IF, LITERAL_SWITCH, LITERAL_SYNCHRONIZED, LITERAL_TRY, LITERAL_WHILE"/>
68
+ </module>
69
+ <module name="RightCurly">
70
+ <property name="option" value="alone"/>
71
+ </module>
72
+ <module name="GenericWhitespace"/>
73
+ <module name="WhitespaceAfter"/>
74
+ <module name="NoWhitespaceBefore"/>
75
+
76
+ <module name="UpperEll"/>
77
+ <module name="DefaultComesLast"/>
78
+ <module name="ArrayTypeStyle"/>
79
+ <module name="MultipleVariableDeclarations"/>
80
+ <module name="ModifierOrder"/>
81
+ <module name="OneStatementPerLine"/>
82
+ <module name="StringLiteralEquality"/>
83
+ <module name="MutableException"/>
84
+ <module name="EqualsHashCode"/>
85
+ <module name="InnerAssignment"/>
86
+ <module name="InterfaceIsType"/>
87
+ <module name="HideUtilityClassConstructor"/>
88
+
89
+ <module name="MemberName"/>
90
+ <module name="LocalVariableName"/>
91
+ <module name="LocalFinalVariableName"/>
92
+ <module name="TypeName"/>
93
+ <module name="PackageName"/>
94
+ <module name="ParameterName"/>
95
+ <module name="StaticVariableName"/>
96
+ <module name="ClassTypeParameterName">
97
+ <property name="format" value="^[A-Z][0-9]?$"/>
98
+ </module>
99
+ <module name="MethodTypeParameterName">
100
+ <property name="format" value="^[A-Z][0-9]?$"/>
101
+ </module>
102
+
103
+ <module name="AvoidStarImport"/>
104
+ <module name="RedundantImport"/>
105
+ <module name="UnusedImports"/>
106
+ <module name="ImportOrder">
107
+ <property name="groups" value="*,javax,java"/>
108
+ <property name="separated" value="true"/>
109
+ <property name="option" value="bottom"/>
110
+ <property name="sortStaticImportsAlphabetically" value="true"/>
111
+ </module>
112
+
113
+ <module name="WhitespaceAround">
114
+ <property name="allowEmptyConstructors" value="true"/>
115
+ <property name="allowEmptyMethods" value="true"/>
116
+ <property name="ignoreEnhancedForColon" value="false"/>
117
+ <property name="tokens" value="
118
+ ASSIGN, BAND, BAND_ASSIGN, BOR, BOR_ASSIGN, BSR, BSR_ASSIGN,
119
+ BXOR, BXOR_ASSIGN, COLON, DIV, DIV_ASSIGN, EQUAL, GE, GT, LAND, LE,
120
+ LITERAL_ASSERT, LITERAL_CATCH, LITERAL_DO, LITERAL_ELSE,
121
+ LITERAL_FINALLY, LITERAL_FOR, LITERAL_IF, LITERAL_RETURN,
122
+ LITERAL_SYNCHRONIZED, LITERAL_TRY, LITERAL_WHILE,
123
+ LOR, LT, MINUS, MINUS_ASSIGN, MOD, MOD_ASSIGN, NOT_EQUAL,
124
+ PLUS, PLUS_ASSIGN, QUESTION, SL, SLIST, SL_ASSIGN, SR, SR_ASSIGN,
125
+ STAR, STAR_ASSIGN, TYPE_EXTENSION_AND"/>
126
+ </module>
127
+ </module>
128
+ </module>
data/config/checkstyle/default.xml ADDED
@@ -0,0 +1,108 @@
1
+ <?xml version="1.0" encoding="UTF-8"?>
2
+ <!DOCTYPE module PUBLIC
3
+ "-//Puppy Crawl//DTD Check Configuration 1.3//EN"
4
+ "http://www.puppycrawl.com/dtds/configuration_1_3.dtd">
5
+ <!--
6
+ This is a subset of ./checkstyle.xml which allows some loose styles
7
+ -->
8
+ <module name="Checker">
9
+ <module name="FileTabCharacter"/>
10
+ <module name="NewlineAtEndOfFile">
11
+ <property name="lineSeparator" value="lf"/>
12
+ </module>
13
+ <module name="RegexpMultiline">
14
+ <property name="format" value="\r"/>
15
+ <property name="message" value="Line contains carriage return"/>
16
+ </module>
17
+ <module name="RegexpMultiline">
18
+ <property name="format" value=" \n"/>
19
+ <property name="message" value="Line has trailing whitespace"/>
20
+ </module>
21
+ <module name="RegexpMultiline">
22
+ <property name="format" value="\n\n\n"/>
23
+ <property name="message" value="Multiple consecutive blank lines"/>
24
+ </module>
25
+ <module name="RegexpMultiline">
26
+ <property name="format" value="\n\n\Z"/>
27
+ <property name="message" value="Blank line before end of file"/>
28
+ </module>
29
+
30
+ <module name="TreeWalker">
31
+ <module name="EmptyBlock">
32
+ <property name="option" value="text"/>
33
+ <property name="tokens" value="
34
+ LITERAL_DO, LITERAL_ELSE, LITERAL_FINALLY, LITERAL_IF,
35
+ LITERAL_FOR, LITERAL_TRY, LITERAL_WHILE, INSTANCE_INIT, STATIC_INIT"/>
36
+ </module>
37
+ <module name="EmptyStatement"/>
38
+ <module name="EmptyForInitializerPad"/>
39
+ <module name="EmptyForIteratorPad">
40
+ <property name="option" value="space"/>
41
+ </module>
42
+ <module name="MethodParamPad">
43
+ <property name="allowLineBreaks" value="true"/>
44
+ <property name="option" value="nospace"/>
45
+ </module>
46
+ <module name="ParenPad"/>
47
+ <module name="TypecastParenPad"/>
48
+ <module name="NeedBraces"/>
49
+ <module name="LeftCurly">
50
+ <property name="option" value="nl"/>
51
+ <property name="tokens" value="CLASS_DEF, CTOR_DEF, INTERFACE_DEF, METHOD_DEF"/>
52
+ </module>
53
+ <module name="LeftCurly">
54
+ <property name="option" value="eol"/>
55
+ <property name="tokens" value="
56
+ LITERAL_CATCH, LITERAL_DO, LITERAL_ELSE, LITERAL_FINALLY, LITERAL_FOR,
57
+ LITERAL_IF, LITERAL_SWITCH, LITERAL_SYNCHRONIZED, LITERAL_TRY, LITERAL_WHILE"/>
58
+ </module>
59
+ <module name="RightCurly">
60
+ <property name="option" value="alone"/>
61
+ </module>
62
+ <module name="GenericWhitespace"/>
63
+ <module name="WhitespaceAfter"/>
64
+ <module name="NoWhitespaceBefore"/>
65
+
66
+ <module name="UpperEll"/>
67
+ <module name="DefaultComesLast"/>
68
+ <module name="ArrayTypeStyle"/>
69
+ <module name="MultipleVariableDeclarations"/>
70
+ <module name="ModifierOrder"/>
71
+ <module name="OneStatementPerLine"/>
72
+ <module name="StringLiteralEquality"/>
73
+ <module name="MutableException"/>
74
+ <module name="EqualsHashCode"/>
75
+ <module name="InnerAssignment"/>
76
+ <module name="InterfaceIsType"/>
77
+ <module name="HideUtilityClassConstructor"/>
78
+
79
+ <module name="MemberName"/>
80
+ <module name="LocalVariableName"/>
81
+ <module name="LocalFinalVariableName"/>
82
+ <module name="TypeName"/>
83
+ <module name="PackageName"/>
84
+ <module name="ParameterName"/>
85
+ <module name="StaticVariableName"/>
86
+ <module name="ClassTypeParameterName">
87
+ <property name="format" value="^[A-Z][0-9]?$"/>
88
+ </module>
89
+ <module name="MethodTypeParameterName">
90
+ <property name="format" value="^[A-Z][0-9]?$"/>
91
+ </module>
92
+
93
+ <module name="WhitespaceAround">
94
+ <property name="allowEmptyConstructors" value="true"/>
95
+ <property name="allowEmptyMethods" value="true"/>
96
+ <property name="ignoreEnhancedForColon" value="false"/>
97
+ <property name="tokens" value="
98
+ ASSIGN, BAND, BAND_ASSIGN, BOR, BOR_ASSIGN, BSR, BSR_ASSIGN,
99
+ BXOR, BXOR_ASSIGN, COLON, DIV, DIV_ASSIGN, EQUAL, GE, GT, LAND, LE,
100
+ LITERAL_ASSERT, LITERAL_CATCH, LITERAL_DO, LITERAL_ELSE,
101
+ LITERAL_FINALLY, LITERAL_FOR, LITERAL_IF, LITERAL_RETURN,
102
+ LITERAL_SYNCHRONIZED, LITERAL_TRY, LITERAL_WHILE,
103
+ LOR, LT, MINUS, MINUS_ASSIGN, MOD, MOD_ASSIGN, NOT_EQUAL,
104
+ PLUS, PLUS_ASSIGN, QUESTION, SL, SLIST, SL_ASSIGN, SR, SR_ASSIGN,
105
+ STAR, STAR_ASSIGN, TYPE_EXTENSION_AND"/>
106
+ </module>
107
+ </module>
108
+ </module>
data/gradle/wrapper/gradle-wrapper.jar CHANGED
Binary file
data/gradle/wrapper/gradle-wrapper.properties CHANGED
@@ -1,6 +1,6 @@
1
- #Tue Aug 11 00:26:20 PDT 2015
1
+ #Wed Jan 13 12:41:02 JST 2016
2
2
  distributionBase=GRADLE_USER_HOME
3
3
  distributionPath=wrapper/dists
4
4
  zipStoreBase=GRADLE_USER_HOME
5
5
  zipStorePath=wrapper/dists
6
- distributionUrl=https\://services.gradle.org/distributions/gradle-2.6-bin.zip
6
+ distributionUrl=https\://services.gradle.org/distributions/gradle-2.10-bin.zip
data/src/main/java/org/embulk/output/BigqueryAuthentication.java CHANGED
@@ -47,9 +47,11 @@ public class BigqueryAuthentication
47
47
 
48
48
  if (authMethod.toLowerCase().equals("compute_engine")) {
49
49
  this.credentials = getComputeCredential();
50
- } else if(authMethod.toLowerCase().equals("json_key")) {
50
+ }
51
+ else if (authMethod.toLowerCase().equals("json_key")) {
51
52
  this.credentials = getServiceAccountCredentialFromJsonFile();
52
- } else {
53
+ }
54
+ else {
53
55
  this.credentials = getServiceAccountCredential();
54
56
  }
55
57
  }
@@ -111,4 +113,4 @@ public class BigqueryAuthentication
111
113
 
112
114
  return client;
113
115
  }
114
- }
116
+ }
data/src/main/java/org/embulk/output/BigqueryOutputPlugin.java CHANGED
@@ -146,7 +146,7 @@ public class BigqueryOutputPlugin
146
146
  }
147
147
 
148
148
  private final Logger log = Exec.getLogger(BigqueryOutputPlugin.class);
149
- private final static String temporaryTableSuffix = Long.toString(System.currentTimeMillis());
149
+ private static final String temporaryTableSuffix = Long.toString(System.currentTimeMillis());
150
150
  private static BigqueryWriter bigQueryWriter;
151
151
 
152
152
  @Override
@@ -161,7 +161,8 @@ public class BigqueryOutputPlugin
161
161
  }
162
162
  try {
163
163
  task.setP12Keyfile(Optional.of(LocalFile.of(task.getP12KeyfilePath().get())));
164
- } catch (IOException ex) {
164
+ }
165
+ catch (IOException ex) {
165
166
  throw Throwables.propagate(ex);
166
167
  }
167
168
  }
@@ -172,7 +173,8 @@ public class BigqueryOutputPlugin
172
173
  }
173
174
  try {
174
175
  task.setSchemaFile(Optional.of(LocalFile.of(task.getSchemaPath().get())));
175
- } catch (IOException ex) {
176
+ }
177
+ catch (IOException ex) {
176
178
  throw Throwables.propagate(ex);
177
179
  }
178
180
  }
@@ -181,7 +183,8 @@ public class BigqueryOutputPlugin
181
183
  if (!task.getJsonKeyfile().isPresent()) {
182
184
  throw new ConfigException("If auth_method is json_key, you have to set json_keyfile");
183
185
  }
184
- } else if (task.getAuthMethod().getString().equals("private_key")) {
186
+ }
187
+ else if (task.getAuthMethod().getString().equals("private_key")) {
185
188
  if (!task.getP12Keyfile().isPresent() || !task.getServiceAccountEmail().isPresent()) {
186
189
  throw new ConfigException("If auth_method is private_key, you have to set both service_account_email and p12_keyfile");
187
190
  }
@@ -200,7 +203,7 @@ public class BigqueryOutputPlugin
200
203
  }
201
204
 
202
205
  try {
203
- bigQueryWriter = new BigqueryWriter.Builder (
206
+ bigQueryWriter = new BigqueryWriter.Builder(
204
207
  task.getAuthMethod().getString(),
205
208
  task.getServiceAccountEmail(),
206
209
  task.getP12Keyfile().transform(localFileToPathString()),
@@ -222,7 +225,8 @@ public class BigqueryOutputPlugin
222
225
 
223
226
  bigQueryWriter.checkConfig(task.getProject(), task.getDataset(), task.getTable());
224
227
 
225
- } catch (IOException | GeneralSecurityException ex) {
228
+ }
229
+ catch (IOException | GeneralSecurityException ex) {
226
230
  throw new ConfigException(ex);
227
231
  }
228
232
  // non-retryable (non-idempotent) output:
@@ -242,7 +246,8 @@ public class BigqueryOutputPlugin
242
246
  if (mode == Mode.delete_in_advance) {
243
247
  try {
244
248
  bigQueryWriter.deleteTable(project, dataset, generateTableName(tableName));
245
- } catch (IOException ex) {
249
+ }
250
+ catch (IOException ex) {
246
251
  log.warn(ex.getMessage());
247
252
  }
248
253
  }
@@ -255,13 +260,16 @@ public class BigqueryOutputPlugin
255
260
  bigQueryWriter.replaceTable(project, dataset, generateTableName(tableName) + "_old", generateTableName(tableName));
256
261
  }
257
262
  bigQueryWriter.replaceTable(project, dataset, generateTableName(tableName), generateTemporaryTableName(tableName));
258
- } catch (TimeoutException | BigqueryWriter.JobFailedException | IOException ex) {
263
+ }
264
+ catch (TimeoutException | BigqueryWriter.JobFailedException | IOException ex) {
259
265
  log.error(ex.getMessage());
260
266
  throw Throwables.propagate(ex);
261
- } finally {
267
+ }
268
+ finally {
262
269
  try {
263
270
  bigQueryWriter.deleteTable(project, dataset, generateTemporaryTableName(tableName));
264
- } catch (IOException ex) {
271
+ }
272
+ catch (IOException ex) {
265
273
  log.warn(ex.getMessage());
266
274
  }
267
275
  }
@@ -328,7 +336,8 @@ public class BigqueryOutputPlugin
328
336
  }
329
337
  log.info(String.format("Writing file [%s]", filePath));
330
338
  output = new BufferedOutputStream(new FileOutputStream(filePath));
331
- } catch (FileNotFoundException ex) {
339
+ }
340
+ catch (FileNotFoundException ex) {
332
341
  throw Throwables.propagate(ex);
333
342
  }
334
343
  fileIndex++;
@@ -339,7 +348,8 @@ public class BigqueryOutputPlugin
339
348
  if (output != null) {
340
349
  try {
341
350
  output.close();
342
- } catch (IOException ex) {
351
+ }
352
+ catch (IOException ex) {
343
353
  throw Throwables.propagate(ex);
344
354
  }
345
355
  }
@@ -349,9 +359,11 @@ public class BigqueryOutputPlugin
349
359
  {
350
360
  try {
351
361
  output.write(buffer.array(), buffer.offset(), buffer.limit());
352
- } catch (IOException ex) {
362
+ }
363
+ catch (IOException ex) {
353
364
  throw Throwables.propagate(ex);
354
- } finally {
365
+ }
366
+ finally {
355
367
  buffer.release();
356
368
  }
357
369
  }
@@ -367,7 +379,8 @@ public class BigqueryOutputPlugin
367
379
  log.info(String.format("Delete local file [%s]", filePath));
368
380
  file.delete();
369
381
  }
370
- } catch (NoSuchAlgorithmException | TimeoutException | BigqueryWriter.JobFailedException | IOException ex) {
382
+ }
383
+ catch (NoSuchAlgorithmException | TimeoutException | BigqueryWriter.JobFailedException | IOException ex) {
371
384
  log.error(ex.getMessage());
372
385
  throw Throwables.propagate(ex);
373
386
  }
@@ -447,15 +460,24 @@ public class BigqueryOutputPlugin
447
460
  append("append"),
448
461
  delete_in_advance("delete_in_advance") {
449
462
  @Override
450
- public boolean isDeleteInAdvance() { return true; }
463
+ public boolean isDeleteInAdvance()
464
+ {
465
+ return true;
466
+ }
451
467
  },
452
468
  replace("replace") {
453
469
  @Override
454
- public boolean isReplaceMode() { return true; }
470
+ public boolean isReplaceMode()
471
+ {
472
+ return true;
473
+ }
455
474
  },
456
475
  replace_backup("replace_backup") {
457
476
  @Override
458
- public boolean isReplaceMode() { return true; }
477
+ public boolean isReplaceMode()
478
+ {
479
+ return true;
480
+ }
459
481
  };
460
482
 
461
483
  private final String string;
@@ -465,8 +487,17 @@ public class BigqueryOutputPlugin
465
487
  this.string = string;
466
488
  }
467
489
 
468
- public String getString() { return string; }
469
- public boolean isReplaceMode() { return false; }
470
- public boolean isDeleteInAdvance() { return true; }
490
+ public String getString()
491
+ {
492
+ return string;
493
+ }
494
+ public boolean isReplaceMode()
495
+ {
496
+ return false;
497
+ }
498
+ public boolean isDeleteInAdvance()
499
+ {
500
+ return true;
501
+ }
471
502
  }
472
503
  }
data/src/main/java/org/embulk/output/BigqueryWriter.java CHANGED
@@ -72,7 +72,8 @@ public class BigqueryWriter
72
72
 
73
73
  if (autoCreateTable) {
74
74
  this.tableSchema = createTableSchema();
75
- } else {
75
+ }
76
+ else {
76
77
  this.tableSchema = null;
77
78
  }
78
79
  }
@@ -100,7 +101,8 @@ public class BigqueryWriter
100
101
  log.info(String.format("Job statistics [%s]", statistics.getLoad()));
101
102
  }
102
103
  return jobStatus;
103
- } catch (IOException ex) {
104
+ }
105
+ catch (IOException ex) {
104
106
  log.warn(ex.getMessage());
105
107
  return "UNKNOWN";
106
108
  }
@@ -118,14 +120,17 @@ public class BigqueryWriter
118
120
  if (jobStatus.equals("DONE")) {
119
121
  log.info(String.format("Job completed successfully. job id:[%s] elapsed_time:%dms status:[%s]", jobRef.getJobId(), elapsedTime, "SUCCESS"));
120
122
  break;
121
- } else if (elapsedTime > jobStatusMaxPollingTime * 1000) {
123
+ }
124
+ else if (elapsedTime > jobStatusMaxPollingTime * 1000) {
122
125
  throw new TimeoutException(String.format("Checking job status...Timeout. job id:[%s] elapsed_time:%dms status:[%s]", jobRef.getJobId(), elapsedTime, "TIMEOUT"));
123
- } else {
126
+ }
127
+ else {
124
128
  log.info(String.format("Checking job status... job id:[%s] elapsed_time:%dms status:[%s]", jobRef.getJobId(), elapsedTime, jobStatus));
125
129
  }
126
130
  Thread.sleep(jobStatusPollingInterval * 1000);
127
131
  }
128
- } catch (InterruptedException ex) {
132
+ }
133
+ catch (InterruptedException ex) {
129
134
  log.warn(ex.getMessage());
130
135
  }
131
136
  }
@@ -171,13 +176,15 @@ public class BigqueryWriter
171
176
 
172
177
  try {
173
178
  jobRef = insert.execute().getJobReference();
174
- } catch (IllegalStateException ex) {
179
+ }
180
+ catch (IllegalStateException ex) {
175
181
  throw new JobFailedException(ex.getMessage());
176
182
  }
177
183
  log.info(String.format("Job executed. job id:[%s] file:[%s]", jobRef.getJobId(), localFilePath));
178
184
  if (isSkipJobResultCheck) {
179
185
  log.info(String.format("Skip job status check. job id:[%s]", jobRef.getJobId()));
180
- } else {
186
+ }
187
+ else {
181
188
  getJobStatusUntilDone(project, jobRef);
182
189
  }
183
190
  }
@@ -203,19 +210,22 @@ public class BigqueryWriter
203
210
 
204
211
  try {
205
212
  jobRef = insert.execute().getJobReference();
206
- } catch (IllegalStateException ex) {
213
+ }
214
+ catch (IllegalStateException ex) {
207
215
  throw new JobFailedException(ex.getMessage());
208
216
  }
209
217
  log.info(String.format("Job executed. job id:[%s]", jobRef.getJobId()));
210
218
  getJobStatusUntilDone(project, jobRef);
211
219
  }
212
220
 
213
- public void deleteTable(String project, String dataset, String table) throws IOException {
221
+ public void deleteTable(String project, String dataset, String table) throws IOException
222
+ {
214
223
  try {
215
224
  Tables.Delete delete = bigQueryClient.tables().delete(project, dataset, table);
216
225
  delete.execute();
217
226
  log.info(String.format("Table deleted. project:%s dataset:%s table:%s", delete.getProjectId(), delete.getDatasetId(), delete.getTableId()));
218
- } catch (GoogleJsonResponseException ex) {
227
+ }
228
+ catch (GoogleJsonResponseException ex) {
219
229
  log.warn(ex.getMessage());
220
230
  }
221
231
  }
@@ -238,7 +248,8 @@ public class BigqueryWriter
238
248
  config.setSchema(tableSchema);
239
249
  config.setCreateDisposition("CREATE_IF_NEEDED");
240
250
  log.info(String.format("table:[%s] will be create if not exists", table));
241
- } else {
251
+ }
252
+ else {
242
253
  config.setCreateDisposition("CREATE_NEVER");
243
254
  }
244
255
  return config;
@@ -252,7 +263,8 @@ public class BigqueryWriter
252
263
 
253
264
  if (append) {
254
265
  config.setWriteDisposition("WRITE_APPEND");
255
- } else {
266
+ }
267
+ else {
256
268
  config.setWriteDisposition("WRITE_TRUNCATE");
257
269
  }
258
270
 
@@ -294,7 +306,8 @@ public class BigqueryWriter
294
306
  ObjectMapper mapper = new ObjectMapper();
295
307
  List<TableFieldSchema> fields = mapper.readValue(stream, new TypeReference<List<TableFieldSchema>>() {});
296
308
  return new TableSchema().setFields(fields);
297
- } finally {
309
+ }
310
+ finally {
298
311
  if (stream != null) {
299
312
  stream.close();
300
313
  }
@@ -306,7 +319,8 @@ public class BigqueryWriter
306
319
  Tables tableRequest = bigQueryClient.tables();
307
320
  try {
308
321
  Table tableData = tableRequest.get(project, dataset, table).execute();
309
- } catch (GoogleJsonResponseException ex) {
322
+ }
323
+ catch (GoogleJsonResponseException ex) {
310
324
  return false;
311
325
  }
312
326
  return true;
@@ -317,13 +331,15 @@ public class BigqueryWriter
317
331
  if (autoCreateTable) {
318
332
  if (!schemaPath.isPresent()) {
319
333
  throw new FileNotFoundException("schema_file is empty");
320
- } else {
334
+ }
335
+ else {
321
336
  File file = new File(schemaPath.orNull());
322
337
  if (!file.exists()) {
323
338
  throw new FileNotFoundException("Can not load schema file.");
324
339
  }
325
340
  }
326
- } else {
341
+ }
342
+ else {
327
343
  if (!isExistTable(project, dataset, table)) {
328
344
  throw new IOException(String.format("table [%s] is not exists", table));
329
345
  }
@@ -347,7 +363,8 @@ public class BigqueryWriter
347
363
 
348
364
  byte[] encoded = (hashedBytes);
349
365
  return new String(encoded);
350
- } finally {
366
+ }
367
+ finally {
351
368
  stream.close();
352
369
  }
353
370
  }
@@ -490,7 +507,8 @@ public class BigqueryWriter
490
507
 
491
508
  public class JobFailedException extends RuntimeException
492
509
  {
493
- public JobFailedException(String message) {
510
+ public JobFailedException(String message)
511
+ {
494
512
  super(message);
495
513
  }
496
514
  }
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: embulk-output-bigquery
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.0
4
+ version: 0.2.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Satoshi Akama
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2016-01-26 00:00:00.000000000 Z
11
+ date: 2016-01-28 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  requirement: !ruby/object:Gem::Requirement
@@ -46,9 +46,11 @@ extensions: []
46
46
  extra_rdoc_files: []
47
47
  files:
48
48
  - .gitignore
49
- - ChangeLog
49
+ - CHANGELOG.md
50
50
  - README.md
51
51
  - build.gradle
52
+ - config/checkstyle/checkstyle.xml
53
+ - config/checkstyle/default.xml
52
54
  - gradle/wrapper/gradle-wrapper.jar
53
55
  - gradle/wrapper/gradle-wrapper.properties
54
56
  - gradlew
@@ -63,7 +65,7 @@ files:
63
65
  - src/test/java/org/embulk/output/TestBigqueryWriter.java
64
66
  - classpath/commons-codec-1.3.jar
65
67
  - classpath/commons-logging-1.1.1.jar
66
- - classpath/embulk-output-bigquery-0.2.0.jar
68
+ - classpath/embulk-output-bigquery-0.2.1.jar
67
69
  - classpath/google-api-client-1.20.0.jar
68
70
  - classpath/google-api-services-bigquery-v2-rev205-1.20.0.jar
69
71
  - classpath/google-http-client-1.20.0.jar
data/ChangeLog DELETED
@@ -1,50 +0,0 @@
1
- Release 0.2.0 - 2016-01-26
2
-
3
- * Added mode parameters and support 4 modes(append, replace, replace_backup, delete_in_advance). @joker1007 thanks!
4
-
5
- Release 0.1.11 - 2015-11-16
6
-
7
- * Change error result display for easy investigation.
8
-
9
- Release 0.1.10 - 2015-10-06
10
-
11
- * Added new auth method - json_keyfile of GCP(Google Cloud Platform)'s service account
12
-
13
- Release 0.1.9 - 2015-08-19
14
-
15
- * Upgraded embulk version to 0.7.1
16
-
17
- Release 0.1.8 - 2015-08-19
18
-
19
- * Supported mapreduce-executor. @frsyuki thanks!
20
- * Fixed job_id generation logic
21
- * Refactored
22
-
23
- Release 0.1.7 - 2015-05-20
24
-
25
- * Added allow_quoted_newlines option
26
- * Upgraded embulk version to 0.6.8
27
-
28
- Release 0.1.6 - 2015-04-23
29
-
30
- * Added ignore_unknown_values option to job_id generation logic.
31
-
32
- Release 0.1.5 - 2015-04-23
33
-
34
- * Added ignore_unknown_values option. @takus thanks!
35
-
36
- Release 0.1.4 - 2015-04-21
37
-
38
- * Added prevent_duplicate_insert option
39
-
40
- Release 0.1.3 - 2015-04-06
41
-
42
- * Added new auth method - pre-defined access token of GCE(Google Compute Engine)
43
- * Updated Google provided libraries
44
- * http-client:google-http-client-jackson2 from 1.19.0 to 1.20.0
45
- * apis:google-api-services-bigquery from v2-rev193-1.19.1 to v2-rev205-1.20.0
46
-
47
- Release 0.1.2 - 2015-04-01
48
-
49
- * Changed bulk-load method from "via GCS" to direct-insert
50
- * added dynamic table creationg option