embulk-output-mysql 0.2.4 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 54ae48d0cd8f8a4b20e470008c338d7f8f50a15f
-  data.tar.gz: 30910e0ea400a868af1e9869ac4337029af5b02a
+  metadata.gz: b35f2d66f92a633b5326a958d7995b73f02f3737
+  data.tar.gz: a059b0f6436a26896cecf2e3bbfb6d305d109405
 SHA512:
-  metadata.gz: d7ec8503e5ecd071de55099962d4bc1937ff49333110577e53310ba1d110a01f1b1c5f7d0659c6802db75fa888a91079f72ad98afd36c8482171531e35233664
-  data.tar.gz: 7db52807e1444a4a17038720fc59ffafa9b9c87b83b83433a0acd693f809746e7a76a325adde92d6e8e79b0fe077e4a050ceb193327aa442a10710d6e35afe67
+  metadata.gz: 9bed384f0d48c7f46bb5bff652b78d2f11080ae569fb2c91229f184c07ef923379b0334f9563d8e3048944358138a8ba6865e14ea0c40033dd6a463846913674
+  data.tar.gz: 03ab7574a43af5fa86e9db38d9d7ad28221182724fc29c9ed5355d259040dcb9bb25f212bd79c2edb27b0361f625ca91ff64c1c8793515dd3dc744a1f7730dac

data/README.md

@@ -1,42 +1,92 @@
-# MySQL output plugins for Embulk
-
-MySQL output plugins for Embulk loads records to MySQL.
-
-## Overview
-
-* **Plugin type**: output
-* **Load all or nothing**: depnds on the mode:
-  * **insert**: no
-  * **replace**: yes
-* **Resume supported**: no
-
-## Configuration
-
-- **host**: database host name (string, required)
-- **port**: database port number (integer, default: 3306)
-- **user**: database login user name (string, required)
-- **password**: database login password (string, default: "")
-- **database**: destination database name (string, required)
-- **table**: destination table name (string, required)
-- **mode**: "replace", "merge" or "insert" (string, required)
-- **batch_size**: size of a single batch insert (integer, default: 16777216)
-- **options**: extra connection properties (hash, default: {})
-
-### Example
-
-```yaml
-out:
-  type: mysql
-  host: localhost
-  user: root
-  password: ""
-  database: my_database
-  table: my_table
-  mode: insert
-```
-
-### Build
-
-```
-$ ./gradlew gem
-```
+# MySQL output plugins for Embulk
+
+MySQL output plugins for Embulk loads records to MySQL.
+
+## Overview
+
+* **Plugin type**: output
+* **Load all or nothing**: depnds on the mode. see bellow.
+* **Resume supported**: depnds on the mode. see bellow.
+
+## Configuration
+
+- **host**: database host name (string, required)
+- **port**: database port number (integer, default: 3306)
+- **user**: database login user name (string, required)
+- **password**: database login password (string, default: "")
+- **database**: destination database name (string, required)
+- **table**: destination table name (string, required)
+- **options**: extra connection properties (hash, default: {})
+- **mode**: "insert", "insert_direct", "truncate_insert", "merge", "merge_direct", or "replace". See bellow. (string, required)
+- **batch_size**: size of a single batch insert (integer, default: 16777216)
+- **default_timezone**: If input column type (embulk type) is timestamp and destination column type is `string` or `nstring`, this plugin needs to format the timestamp into a string. This default_timezone option is used to control the timezone. You can overwrite timezone for each columns using column_options option. (string, default: `UTC`)
+- **column_options**: advanced: a key-value pairs where key is a column name and value is options for the column.
+  - **type**: type of a column when this plugin creates new tables (e.g. `VARCHAR(255)`, `INTEGER NOT NULL UNIQUE`). This used when this plugin creates intermediate tables (insert, insert_truncate and merge modes), when it creates the target table (insert_direct, merge_direct and replace modes), and when it creates nonexistent target table automatically. (string, default: depends on input column type. `BIGINT` if input column type is long, `BOOLEAN` if boolean, `DOUBLE PRECISION` if double, `CLOB` if string, `TIMESTAMP` if timestamp)
+  - **value_type**: This plugin converts input column type (embulk type) into a database type to build a INSERT statement. This value_type option controls the type of the value in a INSERT statement. (string, default: depends on input column type. Available values options are: `byte`, `short`, `int`, `long`, `double`, `float`, `boolean`, `string`, `nstring`, `date`, `time`, `timestamp`, `decimal`, `null`, `pass`)
+  - **timestamp_format**: If input column type (embulk type) is timestamp and value_type is `string` or `nstring`, this plugin needs to format the timestamp value into a string. This timestamp_format option is used to control the format of the timestamp. (string, default: `%Y-%m-%d %H:%M:%S.%6N`)
+  - **timezone**: If input column type (embulk type) is timestamp and value_type is `string` or `nstring`, this plugin needs to format the timestamp value into a string. And if the input column type is timestamp and value_type is `date`, this plugin needs to consider timezone. In those cases, this timezone option is used to control the timezone. (string, value of default_timezone option is used by default)
+
+### Modes
+
+* **insert**:
+  * Behavior: This mode writes rows to some intermediate tables first. If all those tasks run correctly, runs `INSERT INTO <target_table> SELECT * FROM <intermediate_table_1> UNION ALL SELECT * FROM <intermediate_table_2> UNION ALL ...` query.
+  * Transactional: Yes. This mode successfully writes all rows, or fails with writing zero rows.
+  * Resumable: Yes.
+* **insert_direct**:
+  * Behavior: This mode inserts rows to the target table directly.
+  * Transactional: No. If fails, the target table could have some rows inserted.
+  * Resumable: No.
+* **truncate_insert**:
+  * Behavior: Same with `insert` mode excepting that it truncates the target table right before the last `INSERT ...` query.
+  * Transactional: Yes.
+  * Resumable: Yes.
+* **merge**:
+  * Behavior: This mode writes rows to some intermediate tables first. If all those tasks run correctly, runs `INSERT INTO <target_table> SELECT * FROM <intermediate_table_1> UNION ALL SELECT * FROM <intermediate_table_2> UNION ALL ... ON DUPLICATE KEY UPDATE ...` query.
+  * Transactional: Yes.
+  * Resumable: Yes.
+* **merge_direct**:
+  * Behavior: This mode inserts rows to the target table directory using `INSERT INTO ... ON DUPLICATE KEY UPDATE ...` query.
+  * Transactional: No.
+  * Resumable: No.
+* **replace**:
+  * Behavior: Same with `insert` mode excepting that it truncates the target table right before the last `INSERT ...` query.
+  * Transactional: Yes.
+  * Resumable: No.
+
+### Example
+
+```yaml
+out:
+  type: mysql
+  host: localhost
+  user: root
+  password: ""
+  database: my_database
+  table: my_table
+  mode: insert
+```
+
+Advanced configuration:
+
+```yaml
+out:
+  type: mysql
+  host: localhost
+  user: root
+  password: ""
+  database: my_database
+  table: my_table
+  options: {connectTimeout: 20000}
+  mode: insert_direct
+  column_options:
+    my_col_1: {type: 'TEXT'}
+    my_col_3: {type: 'INT NOT NULL'}
+    my_col_4: {value_type: string, timestamp_format: `%Y-%m-%d %H:%M:%S %z`, timezone: '-0700'}
+    my_col_5: {type: 'DECIMAL(18,9)', value_type: pass}
+```
+
+### Build
+
+```
+$ ./gradlew gem
+```

data/build.gradle

@@ -1,7 +1,7 @@
-dependencies {
-    compile project(':embulk-output-jdbc')
-
-    compile 'mysql:mysql-connector-java:5.1.34'
-
-    testCompile project(':embulk-output-jdbc').sourceSets.test.output
-}
+dependencies {
+    compile project(':embulk-output-jdbc')
+
+    compile 'mysql:mysql-connector-java:5.1.34'
+
+    testCompile project(':embulk-output-jdbc').sourceSets.test.output
+}

data/lib/embulk/output/mysql.rb

@@ -1,3 +1,3 @@
-Embulk::JavaPlugin.register_output(
-  :mysql, "org.embulk.output.MySQLOutputPlugin",
-  File.expand_path('../../../../classpath', __FILE__))
+Embulk::JavaPlugin.register_output(
+  :mysql, "org.embulk.output.MySQLOutputPlugin",
+  File.expand_path('../../../../classpath', __FILE__))

data/src/main/java/org/embulk/output/MySQLOutputPlugin.java

@@ -1,10 +1,10 @@
 package org.embulk.output;
 
+import java.util.List;
 import java.util.Properties;
 import java.io.IOException;
 import java.sql.SQLException;
-
-import org.embulk.output.mysql.MySQLBatchUpsert;
+import com.google.common.base.Optional;
 import org.embulk.config.Config;
 import org.embulk.config.ConfigDefault;
 import org.embulk.output.jdbc.AbstractJdbcOutputPlugin;
@@ -42,6 +42,14 @@ public class MySQLOutputPlugin
         return MySQLPluginTask.class;
     }
 
+    @Override
+    protected Features getFeatures(PluginTask task)
+    {
+        return new Features()
+            .setMaxTableNameLength(64)
+            .setIgnoreMergeKeys(true);
+    }
+
     @Override
     protected MySQLOutputConnector getConnector(PluginTask task, boolean retryableMetadataOperation)
     {
@@ -51,8 +59,6 @@ public class MySQLOutputPlugin
                 t.getHost(), t.getPort(), t.getDatabase());
 
         Properties props = new Properties();
-        props.setProperty("user", t.getUser());
-        props.setProperty("password", t.getPassword());
 
         props.setProperty("rewriteBatchedStatements", "true");
         props.setProperty("useCompression", "true");
@@ -88,13 +94,18 @@ public class MySQLOutputPlugin
 
         props.putAll(t.getOptions());
 
+        // TODO validate task.getMergeKeys is null
+
+        props.setProperty("user", t.getUser());
+        logger.info("Connecting to {} options {}", url, props);
+        props.setProperty("password", t.getPassword());
+
         return new MySQLOutputConnector(url, props);
     }
 
     @Override
-    protected BatchInsert newBatchInsert(PluginTask task) throws IOException, SQLException
+    protected BatchInsert newBatchInsert(PluginTask task, Optional<List<String>> mergeKeys) throws IOException, SQLException
     {
-        MySQLOutputConnector connector = getConnector(task, true);
-        return task.getMode().isMerge() ? new MySQLBatchUpsert(connector) : new MySQLBatchInsert(connector);
+        return new MySQLBatchInsert(getConnector(task, true), mergeKeys);
     }
 }

data/src/main/java/org/embulk/output/mysql/MySQLBatchInsert.java

@@ -1,36 +1,38 @@
-package org.embulk.output.mysql;
-
-import java.io.IOException;
-import java.sql.Types;
-import java.sql.PreparedStatement;
-import java.sql.SQLException;
-import org.embulk.output.jdbc.StandardBatchInsert;
-
-public class MySQLBatchInsert
-        extends StandardBatchInsert
-{
-    public MySQLBatchInsert(MySQLOutputConnector connector) throws IOException, SQLException
-    {
-        super(connector);
-    }
-
-    @Override
-    public void setFloat(float v) throws IOException, SQLException
-    {
-        if (Float.isNaN(v) || Float.isInfinite(v)) {
-            setNull(Types.REAL);  // TODO get through argument
-        } else {
-            super.setFloat(v);
-        }
-    }
-
-    @Override
-    public void setDouble(double v) throws IOException, SQLException
-    {
-        if (Double.isNaN(v) || Double.isInfinite(v)) {
-            setNull(Types.DOUBLE);  // TODO get through argument
-        } else {
-            super.setDouble(v);
-        }
-    }
-}
+package org.embulk.output.mysql;
+
+import java.util.List;
+import java.io.IOException;
+import java.sql.Types;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import com.google.common.base.Optional;
+import org.embulk.output.jdbc.StandardBatchInsert;
+
+public class MySQLBatchInsert
+        extends StandardBatchInsert
+{
+    public MySQLBatchInsert(MySQLOutputConnector connector, Optional<List<String>> mergeKeys) throws IOException, SQLException
+    {
+        super(connector, mergeKeys);
+    }
+
+    @Override
+    public void setFloat(float v) throws IOException, SQLException
+    {
+        if (Float.isNaN(v) || Float.isInfinite(v)) {
+            setNull(Types.REAL);  // TODO get through argument
+        } else {
+            super.setFloat(v);
+        }
+    }
+
+    @Override
+    public void setDouble(double v) throws IOException, SQLException
+    {
+        if (Double.isNaN(v) || Double.isInfinite(v)) {
+            setNull(Types.DOUBLE);  // TODO get through argument
+        } else {
+            super.setDouble(v);
+        }
+    }
+}

data/src/main/java/org/embulk/output/mysql/MySQLOutputConnection.java

@@ -1,8 +1,9 @@
 package org.embulk.output.mysql;
 
+import java.util.List;
 import java.sql.Connection;
 import java.sql.SQLException;
-
+import org.embulk.output.jdbc.JdbcColumn;
 import org.embulk.output.jdbc.JdbcSchema;
 import org.embulk.output.jdbc.JdbcOutputConnection;
 
@@ -17,13 +18,12 @@ public class MySQLOutputConnection
     }
 
     @Override
-    protected String buildPrepareUpsertSql(String toTable, JdbcSchema toTableSchema) throws SQLException
+    protected String buildPreparedMergeSql(String toTable, JdbcSchema toTableSchema, List<String> mergeKeys) throws SQLException
     {
         StringBuilder sb = new StringBuilder();
 
         sb.append("INSERT INTO ");
         quoteIdentifierString(sb, toTable);
-
         sb.append(" (");
         for (int i=0; i < toTableSchema.getCount(); i++) {
             if(i != 0) { sb.append(", "); }
@@ -35,11 +35,43 @@ public class MySQLOutputConnection
             sb.append("?");
         }
         sb.append(")");
-
         sb.append(" ON DUPLICATE KEY UPDATE ");
         for (int i=0; i < toTableSchema.getCount(); i++) {
             if(i != 0) { sb.append(", "); }
-            final String columnName = quoteIdentifierString(toTableSchema.getColumnName(i));
+            String columnName = quoteIdentifierString(toTableSchema.getColumnName(i));
+            sb.append(columnName).append(" = VALUES(").append(columnName).append(")");
+        }
+
+        return sb.toString();
+    }
+
+    @Override
+    protected String buildCollectMergeSql(List<String> fromTables, JdbcSchema schema, String toTable, List<String> mergeKeys) throws SQLException
+    {
+        StringBuilder sb = new StringBuilder();
+
+        sb.append("INSERT INTO ");
+        quoteIdentifierString(sb, toTable);
+        sb.append(" (");
+        for (int i=0; i < schema.getCount(); i++) {
+            if (i != 0) { sb.append(", "); }
+            quoteIdentifierString(sb, schema.getColumnName(i));
+        }
+        sb.append(") ");
+        for (int i=0; i < fromTables.size(); i++) {
+            if (i != 0) { sb.append(" UNION ALL "); }
+            sb.append("SELECT ");
+            for (int j=0; j < schema.getCount(); j++) {
+                if (j != 0) { sb.append(", "); }
+                quoteIdentifierString(sb, schema.getColumnName(j));
+            }
+            sb.append(" FROM ");
+            quoteIdentifierString(sb, fromTables.get(i));
+        }
+        sb.append(" ON DUPLICATE KEY UPDATE ");
+        for (int i=0; i < schema.getCount(); i++) {
+            if(i != 0) { sb.append(", "); }
+            String columnName = quoteIdentifierString(schema.getColumnName(i));
             sb.append(columnName).append(" = VALUES(").append(columnName).append(")");
         }
 
@@ -47,13 +79,13 @@ public class MySQLOutputConnection
     }
 
     @Override
-    protected String convertTypeName(String typeName)
+    protected String buildColumnTypeName(JdbcColumn c)
     {
-        switch(typeName) {
+        switch(c.getSimpleTypeName()) {
         case "CLOB":
             return "TEXT";
         default:
-            return typeName;
+            return super.buildColumnTypeName(c);
         }
     }
 }

data/src/main/java/org/embulk/output/mysql/MySQLOutputConnector.java

@@ -1,42 +1,42 @@
-package org.embulk.output.mysql;
-
-import java.util.Properties;
-import java.sql.Driver;
-import java.sql.Connection;
-import java.sql.SQLException;
-import org.embulk.output.jdbc.JdbcOutputConnector;
-import org.embulk.output.jdbc.JdbcOutputConnection;
-
-public class MySQLOutputConnector
-        implements JdbcOutputConnector
-{
-    private final Driver driver;
-    private final String url;
-    private final Properties properties;
-
-    public MySQLOutputConnector(String url, Properties properties)
-    {
-        try {
-            this.driver = new com.mysql.jdbc.Driver();  // new com.mysql.jdbc.Driver throws SQLException
-        } catch (SQLException ex) {
-            throw new RuntimeException(ex);
-        }
-        this.url = url;
-        this.properties = properties;
-    }
-
-    @Override
-    public MySQLOutputConnection connect(boolean autoCommit) throws SQLException
-    {
-        Connection c = driver.connect(url, properties);
-        try {
-            MySQLOutputConnection con = new MySQLOutputConnection(c, autoCommit);
-            c = null;
-            return con;
-        } finally {
-            if (c != null) {
-                c.close();
-            }
-        }
-    }
-}
+package org.embulk.output.mysql;
+
+import java.util.Properties;
+import java.sql.Driver;
+import java.sql.Connection;
+import java.sql.SQLException;
+import org.embulk.output.jdbc.JdbcOutputConnector;
+import org.embulk.output.jdbc.JdbcOutputConnection;
+
+public class MySQLOutputConnector
+        implements JdbcOutputConnector
+{
+    private final Driver driver;
+    private final String url;
+    private final Properties properties;
+
+    public MySQLOutputConnector(String url, Properties properties)
+    {
+        try {
+            this.driver = new com.mysql.jdbc.Driver();  // new com.mysql.jdbc.Driver throws SQLException
+        } catch (SQLException ex) {
+            throw new RuntimeException(ex);
+        }
+        this.url = url;
+        this.properties = properties;
+    }
+
+    @Override
+    public MySQLOutputConnection connect(boolean autoCommit) throws SQLException
+    {
+        Connection c = driver.connect(url, properties);
+        try {
+            MySQLOutputConnection con = new MySQLOutputConnection(c, autoCommit);
+            c = null;
+            return con;
+        } finally {
+            if (c != null) {
+                c.close();
+            }
+        }
+    }
+}

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: embulk-output-mysql
 version: !ruby/object:Gem::Version
-  version: 0.2.4
+  version: 0.3.0
 platform: ruby
 authors:
 - Sadayuki Furuhashi
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2015-05-12 00:00:00.000000000 Z
+date: 2015-05-19 00:00:00.000000000 Z
 dependencies: []
 description: Inserts or updates records to a table.
 email:
@@ -22,11 +22,10 @@ files:
 - lib/embulk/output/mysql.rb
 - src/main/java/org/embulk/output/MySQLOutputPlugin.java
 - src/main/java/org/embulk/output/mysql/MySQLBatchInsert.java
-- src/main/java/org/embulk/output/mysql/MySQLBatchUpsert.java
 - src/main/java/org/embulk/output/mysql/MySQLOutputConnection.java
 - src/main/java/org/embulk/output/mysql/MySQLOutputConnector.java
-- classpath/embulk-output-jdbc-0.2.4.jar
-- classpath/embulk-output-mysql-0.2.4.jar
+- classpath/embulk-output-jdbc-0.3.0.jar
+- classpath/embulk-output-mysql-0.3.0.jar
 - classpath/mysql-connector-java-5.1.34.jar
 homepage: https://github.com/embulk/embulk-output-jdbc
 licenses:

data/src/main/java/org/embulk/output/mysql/MySQLBatchUpsert.java

@@ -1,22 +0,0 @@
-package org.embulk.output.mysql;
-
-import org.embulk.output.jdbc.JdbcOutputConnection;
-import org.embulk.output.jdbc.JdbcSchema;
-import java.io.IOException;
-import java.sql.PreparedStatement;
-import java.sql.SQLException;
-
-public class MySQLBatchUpsert extends MySQLBatchInsert {
-
-    public MySQLBatchUpsert(MySQLOutputConnector connector) throws IOException, SQLException {
-        super(connector);
-    }
-
-    @Override
-    protected PreparedStatement newPreparedStatement(JdbcOutputConnection connection,
-                                                     String loadTable, JdbcSchema insertSchema) throws SQLException
-    {
-        return connection.prepareUpsertSql(loadTable, insertSchema);
-    }
-
-}