embulk-input-mysql 0.9.3 → 0.10.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 744e51df72489a3bf53272c5df5d459ba838489e
-  data.tar.gz: 9df128d4fa0b33718b3db3c6407a184bc3f699b8
+SHA256:
+  metadata.gz: cc441427089a73c25d30442c2683f881bebcb3d6297d3ef062ebb3a069912a49
+  data.tar.gz: c9dba75abd31ef6fff03aa0e7a5fb3938e8df75ef3202ddfedebf106419e4902
 SHA512:
-  metadata.gz: 5951714019851189f8992454620fcbeb8ed070aabf8051f480764a9b39dfbaa8fa73dcd2eb1f4442dab9d6b428363881cce9ca50f67c5ed435ca7ab63a646e38
-  data.tar.gz: ca3690c6eed9cff67728fd99e65afe21a57a74e05919a9fb72c8538e7b4de7136777b90a05793d60098d8dbfdeabbb91cefbab712a15ddb86b139f1d06b5aea5
+  metadata.gz: 7105db953dedce4d4b84bffcdb9219cdc52dd889592a6907a3b3c2b5ca52d821ec29029d2f9765bbce29779f1fd0df193adc2c86029eb08dcde84065db697fe9
+  data.tar.gz: 75a2c458dfb3b43d0d0321b145c5d8db6789d9a3408110d3a807e358f9f15d40e032cb5f9e6e20258f0ac0f8b51be8921739770c098805a2d441e8cb2ca19948

data/README.md

@@ -7,6 +7,10 @@ MySQL input plugin for Embulk loads records from MySQL.
 * **Plugin type**: input
 * **Resume supported**: yes
 
+**[WARNING!]**
+
+- The default embulk type for MySQL JSON type will be changed from `string` to `json`.
+
 ## Configuration
 
 - **driver_path**: path to the jar file of the MySQL JDBC driver. If not set, the bundled JDBC driver (MySQL Connector/J 5.1.44) will be used (string). NOTE: embulk-input-mysql 0.9.0 upgraded the bundled MySQL Connector/J version from 5.1.34 to 5.1.44 . And set useLegacyDatetimeCode=false by default in order to get correct datetime value when the server timezone and the client timezone are different. Set useLegacyDatetimeCode=true if you need to get datetime value same as older embulk-input-mysql.
@@ -41,7 +45,8 @@ MySQL input plugin for Embulk loads records from MySQL.
 - **last_record**: values of the last record for incremental loading (array of objects, default: load all records)
 - **default_timezone**: If the sql type of a column is `date`/`time`/`datetime` and the embulk type is `string`, column values are formatted int this default_timezone. You can overwrite timezone for each columns using column_options option. (string, default: `UTC`)
 - **use_legacy_datetime_code**: recommended not to set the property (boolean, default: false). If true, embulk-output-mysql will get wrong datetime values when the server timezone and the client server timezone are different as older embulk-output-mysql did.
-- **column_options**: advanced: a key-value pairs where key is a column name and value is options for the column.
+- **default_column_options**: advanced: column_options for each JDBC type as default. key-value pairs where key is a JDBC type (e.g. 'DATE', 'BIGINT') and value is same as column_options's value.
+- **column_options**: advanced: key-value pairs where key is a column name and value is options for the column.
   - **value_type**: embulk get values from database as this value_type. Typically, the value_type determines `getXXX` method of `java.sql.PreparedStatement`.
   (string, default: depends on the sql type of the column. Available values options are: `long`, `double`, `float`, `decimal`, `boolean`, `string`, `json`, `date`, `time`, `timestamp`)
   - **type**: Column values are converted to this embulk type.
@@ -50,6 +55,8 @@ MySQL input plugin for Embulk loads records from MySQL.
   - **timestamp_format**: If the sql type of the column is `date`/`time`/`datetime` and the embulk type is `string`, column values are formatted by this timestamp_format. And if the embulk type is `timestamp`, this timestamp_format may be used in the output plugin. For example, stdout plugin use the timestamp_format, but *csv formatter plugin doesn't use*. (string, default : `%Y-%m-%d` for `date`, `%H:%M:%S` for `time`, `%Y-%m-%d %H:%M:%S` for `timestamp`)
   - **timezone**: If the sql type of the column is `date`/`time`/`datetime` and the embulk type is `string`, column values are formatted in this timezone.
 (string, value of default_timezone option is used by default)
+- **before_setup**: if set, this SQL will be executed before setup. You can prepare table for input by this option.
+- **before_select**: if set, this SQL will be executed before the SELECT query in the same transaction.
 - **after_select**: if set, this SQL will be executed after the SELECT query in the same transaction.
 
 
@@ -86,7 +93,7 @@ Then, it updates `last_record: ` so that next execution uses the updated last_re
 CREATE INDEX embulk_incremental_loading_index ON table (updated_at, id);
 ```
 
-Recommended usage is to leave `incremental_columns` unset and let this plugin automatically finds an AUTO_INCREMENT primary key. Currently, only strings and integers are supported as incremental_columns.
+Recommended usage is to leave `incremental_columns` unset and let this plugin automatically finds an AUTO_INCREMENT primary key. Currently, only strings, integers, DATETIME and TIMESTAMP are supported as incremental_columns.
 
 
 ## Trouble shooting
@@ -178,6 +185,9 @@ in:
   table: "my_table"
   select: "col1, col2, col3"
   where: "col4 != 'a'"
+  default_column_options:
+    TIMESTAMP: { type: string, timestamp_format: "%Y/%m/%d %H:%M:%S", timezone: "+0900"}
+    BIGINT: { type: string }
   column_options:
     col1: {type: long}
     col3: {type: string, timestamp_format: "%Y/%m/%d", timezone: "+0900"}

data/build.gradle

@@ -4,5 +4,5 @@ dependencies {
     compile 'mysql:mysql-connector-java:5.1.44'
     defaultJdbcDriver 'mysql:mysql-connector-java:5.1.44'
 
-    testCompile 'org.embulk:embulk-standards:0.8.15'
+    testCompile 'org.embulk:embulk-standards:0.9.12'
 }

data/src/main/java/org/embulk/input/mysql/getter/MySQLColumnGetterFactory.java

@@ -1,6 +1,5 @@
 package org.embulk.input.mysql.getter;
 
-import com.google.common.base.Optional;
 import org.embulk.config.ConfigException;
 import org.embulk.input.jdbc.AbstractJdbcInputPlugin.PluginTask;
 import org.embulk.input.jdbc.JdbcColumn;
@@ -8,8 +7,10 @@ import org.embulk.input.jdbc.JdbcColumnOption;
 import org.embulk.input.jdbc.JdbcInputConnection;
 import org.embulk.input.jdbc.getter.ColumnGetter;
 import org.embulk.input.jdbc.getter.ColumnGetterFactory;
+import org.embulk.input.jdbc.getter.JsonColumnGetter;
 import org.embulk.input.mysql.MySQLInputConnection;
 import org.embulk.spi.PageBuilder;
+import org.embulk.spi.type.Types;
 import org.joda.time.DateTimeZone;
 
 import java.util.TimeZone;
@@ -54,8 +55,19 @@ public class MySQLColumnGetterFactory
             else { // TIMESTAMP
                 return new MySQLTimestampTimestampIncrementalHandler(sessionTimeZone, getter);
             }
+        case "JSON":
+            return new JsonColumnGetter(to, Types.JSON);
         default:
             return getter;
         }
     }
+
+    @Override
+    protected String sqlTypeToValueType(JdbcColumn column, int sqlType)
+    {
+        if ("json".equals(column.getTypeName())) {
+            return "json";
+        }
+        return super.sqlTypeToValueType(column, sqlType);
+    }
 }

data/src/test/java/org/embulk/input/mysql/BasicTest.java

@@ -134,4 +134,13 @@ public class BasicTest
         assertThat(readSortedFile(out1), is(readResource("test_valuetype_decimal_expected.csv")));
         assertThat(result1.getConfigDiff(), is((ConfigDiff) loadYamlResource(embulk, "test_valuetype_decimal_expected.diff")));
     }
+
+    @Test
+    public void testValueTypeJson() throws Exception
+    {
+        Path out1 = embulk.createTempFile("csv");
+        TestingEmbulk.RunResult result1 = embulk.runInput(baseConfig.merge(loadYamlResource(embulk, "test_valuetype_json_config.yml")), out1);
+        assertThat(readSortedFile(out1), is(readResource("test_valuetype_json_expected.csv")));
+        assertThat(result1.getConfigDiff(), is((ConfigDiff) loadYamlResource(embulk, "test_valuetype_json_expected.diff")));
+    }
 }

data/src/test/java/org/embulk/input/mysql/MySQLColumnGetterFactoryTest.java

@@ -0,0 +1,70 @@
+package org.embulk.input.mysql;
+
+import com.google.common.base.Optional;
+import org.embulk.config.TaskSource;
+import org.embulk.input.jdbc.JdbcColumn;
+import org.embulk.input.jdbc.JdbcColumnOption;
+import org.embulk.input.jdbc.getter.ColumnGetterFactory;
+import org.embulk.input.jdbc.getter.JsonColumnGetter;
+import org.embulk.input.jdbc.getter.StringColumnGetter;
+import org.embulk.input.mysql.getter.MySQLColumnGetterFactory;
+import org.embulk.spi.time.TimestampFormat;
+import org.embulk.spi.type.Type;
+import org.joda.time.DateTimeZone;
+import org.junit.Test;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.IsInstanceOf.instanceOf;
+
+public class MySQLColumnGetterFactoryTest
+{
+    @Test
+    public void testJsonSupported()
+    {
+        JdbcColumnOption mockColumnOption = new JdbcColumnOption()
+        {
+            @Override
+            public String getValueType()
+            {
+                return "coalesce";
+            }
+
+            @Override
+            public Optional<Type> getType()
+            {
+                return Optional.absent();
+            }
+
+            @Override
+            public Optional<TimestampFormat> getTimestampFormat()
+            {
+                return Optional.absent();
+            }
+
+            @Override
+            public Optional<DateTimeZone> getTimeZone()
+            {
+                return Optional.absent();
+            }
+
+            @Override
+            public void validate()
+            {
+
+            }
+
+            @Override
+            public TaskSource dump()
+            {
+                return null;
+            }
+        };
+        JdbcColumn jsonColumn = new JdbcColumn("any_column_name", "JSON", 1, 0, 0); // 1: CHAR
+
+        ColumnGetterFactory fac = new MySQLColumnGetterFactory(null, null);
+        assertThat(fac.newColumnGetter(null, null, jsonColumn, mockColumnOption), instanceOf(JsonColumnGetter.class));
+
+        fac = new ColumnGetterFactory(null, null);
+        assertThat(fac.newColumnGetter(null, null, jsonColumn, mockColumnOption), instanceOf(StringColumnGetter.class));
+    }
+}

data/src/test/java/org/embulk/input/mysql/MySQLTests.java

@@ -3,11 +3,14 @@ package org.embulk.input.mysql;
 import com.google.common.base.Throwables;
 import com.google.common.collect.ImmutableList;
 import com.google.common.io.ByteStreams;
+
 import org.embulk.config.ConfigSource;
 import org.embulk.test.EmbulkTests;
 
 import java.io.IOException;
 
+import jnr.ffi.Platform;
+import jnr.ffi.Platform.OS;
 import static java.util.Locale.ENGLISH;
 
 public class MySQLTests
@@ -35,7 +38,7 @@ public class MySQLTests
                 .add(config.get(String.class, "port", "3306"))
                 .add(config.get(String.class, "database"))
                 .add("-e")
-                .add(sql);
+                .add(convert(sql));
 
         ProcessBuilder pb = new ProcessBuilder(args.build());
         pb.redirectErrorStream(true);
@@ -52,4 +55,13 @@ public class MySQLTests
                     "Command finished with non-zero exit code. Exit code is %d.", code));
         }
     }
+
+    private static String convert(String sql)
+    {
+        if (Platform.getNativePlatform().getOS().equals(OS.WINDOWS)) {
+            // '"' should be '\"' and '\' should be '\\' in Windows
+            return sql.replace("\\\\", "\\").replace("\\", "\\\\").replace("\"", "\\\"");
+        }
+        return sql;
+    }
 }

data/src/test/java/org/embulk/input/mysql/OptionTest.java

@@ -0,0 +1,93 @@
+package org.embulk.input.mysql;
+
+import org.embulk.config.ConfigSource;
+import org.embulk.input.MySQLInputPlugin;
+import org.embulk.spi.InputPlugin;
+import org.embulk.test.EmbulkTests;
+import org.embulk.test.TestingEmbulk;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+
+import java.nio.file.Path;
+
+import static org.embulk.input.mysql.MySQLTests.execute;
+import static org.embulk.test.EmbulkTests.readSortedFile;
+import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertThat;
+
+public class OptionTest
+{
+    private static final String BASIC_RESOURCE_PATH = "org/embulk/input/mysql/test/expect/option/";
+
+    private static ConfigSource loadYamlResource(TestingEmbulk embulk, String fileName)
+    {
+        return embulk.loadYamlResource(BASIC_RESOURCE_PATH + fileName);
+    }
+
+    private static String readResource(String fileName)
+    {
+        return EmbulkTests.readResource(BASIC_RESOURCE_PATH + fileName);
+    }
+
+    @Rule
+    public TestingEmbulk embulk = TestingEmbulk.builder()
+            .registerPlugin(InputPlugin.class, "mysql", MySQLInputPlugin.class)
+            .build();
+
+    private ConfigSource baseConfig;
+
+    @Before
+    public void setup()
+    {
+        baseConfig = MySQLTests.baseConfig();
+        execute(readResource("setup.sql")); // setup rows
+    }
+
+    @Test
+    public void testBeforeSetup() throws Exception
+    {
+        Path out1 = embulk.createTempFile("csv");
+        TestingEmbulk.RunResult result1 = embulk.runInput(baseConfig.merge(loadYamlResource(embulk, "before_setup.yml")), out1);
+        assertThat(readSortedFile(out1), is(readResource("before_setup_expected.csv")));
+    }
+
+    @Test
+    public void testBeforeSelect() throws Exception
+    {
+        Path out1 = embulk.createTempFile("csv");
+        TestingEmbulk.RunResult result1 = embulk.runInput(baseConfig.merge(loadYamlResource(embulk, "before_select.yml")), out1);
+        assertThat(readSortedFile(out1), is(readResource("before_select_expected.csv")));
+    }
+
+    @Test
+    public void testAfterSelect() throws Exception
+    {
+        Path out1 = embulk.createTempFile("csv");
+        TestingEmbulk.RunResult result1 = embulk.runInput(baseConfig.merge(loadYamlResource(embulk, "after_select.yml")), out1);
+        assertThat(readSortedFile(out1), is(readResource("expected1.csv")));
+
+        baseConfig = MySQLTests.baseConfig();
+        Path out2 = embulk.createTempFile("csv");
+        TestingEmbulk.RunResult result2 = embulk.runInput(baseConfig.merge(loadYamlResource(embulk, "input.yml")), out2);
+        assertThat(readSortedFile(out2), is(readResource("after_select_expected.csv")));
+    }
+
+    @Test
+    public void testColumnOptions() throws Exception
+    {
+        Path out1 = embulk.createTempFile("csv");
+        TestingEmbulk.RunResult result1 = embulk.runInput(baseConfig.merge(loadYamlResource(embulk, "column_options.yml")), out1);
+        assertThat(readSortedFile(out1), is(readResource("column_options_expected.csv")));
+    }
+
+    @Test
+    public void testDefaultColumnOptions() throws Exception
+    {
+        Path out1 = embulk.createTempFile("csv");
+        TestingEmbulk.RunResult result1 = embulk.runInput(baseConfig.merge(loadYamlResource(embulk, "default_column_options.yml")), out1);
+        assertThat(readSortedFile(out1), is(readResource("default_column_options_expected.csv")));
+    }
+
+
+}

data/src/test/resources/org/embulk/input/mysql/test/expect/basic/setup.sql

@@ -17,6 +17,7 @@ create table test1 (
     c13 timestamp,
     c14 time,
     c15 datetime(6),
+    c16 json,
     primary key(id)
 );
 
@@ -36,6 +37,7 @@ insert into test1 values(
     null,
     '2015-06-04 23:45:06',
     null,
+    null,
     null
 );
 
@@ -55,7 +57,8 @@ insert into test1 values(
     '2015-06-04 12:34:56',
     '2015-06-04 23:45:06',
     '08:04:02',
-    '2015-06-04 01:02:03.123456'
+    '2015-06-04 01:02:03.123456',
+    '{"key":"value"}'
 );
 
 drop table if exists test2;

data/src/test/resources/org/embulk/input/mysql/test/expect/basic/test_boolean_expected.csv

@@ -1,2 +1,2 @@
-10,,,,,,,,,,,,,2015-06-04 23:45:06.000000 +0000,,
-11,true,true,false,false,true,true,false,true,,,2015-06-03 21:00:00.000000 +0000,2015-06-04 12:34:56.000000 +0000,2015-06-04 23:45:06.000000 +0000,1970-01-01 06:04:02.000000 +0000,2015-06-04 01:02:03.123456 +0000
+10,,,,,,,,,,,,,2015-06-04 23:45:06.000000 +0000,,,
+11,true,true,false,false,true,true,false,true,,,2015-06-03 21:00:00.000000 +0000,2015-06-04 12:34:56.000000 +0000,2015-06-04 23:45:06.000000 +0000,1970-01-01 06:04:02.000000 +0000,2015-06-04 01:02:03.123456 +0000,"{""key"":""value""}"

data/src/test/resources/org/embulk/input/mysql/test/expect/basic/test_double_expected.csv

@@ -1,2 +1,2 @@
-10,,,,,,,,,,,,,2015-06-04 23:45:06.000000 +0000,,
-11,99.0,9999.0,-9.9999999E7,-1.0E16,1.2345000505447388,1.234567890123,-1234.0,1.2345678901234568E17,5678.0,,2015-06-03 21:00:00.000000 +0000,2015-06-04 12:34:56.000000 +0000,2015-06-04 23:45:06.000000 +0000,1970-01-01 06:04:02.000000 +0000,2015-06-04 01:02:03.123456 +0000
+10,,,,,,,,,,,,,2015-06-04 23:45:06.000000 +0000,,,
+11,99.0,9999.0,-9.9999999E7,-1.0E16,1.2345000505447388,1.234567890123,-1234.0,1.2345678901234568E17,5678.0,,2015-06-03 21:00:00.000000 +0000,2015-06-04 12:34:56.000000 +0000,2015-06-04 23:45:06.000000 +0000,1970-01-01 06:04:02.000000 +0000,2015-06-04 01:02:03.123456 +0000,"{""key"":""value""}"

data/src/test/resources/org/embulk/input/mysql/test/expect/basic/test_expected.csv

@@ -1,2 +1,2 @@
-10,,,,,,,,,,,,,2015-06-04 23:45:06.000000 +0000,,
-11,99,9999,-99999999,-9999999999999999,1.2345000505447388,1.234567890123,-1234.0,1.2345678901234568E17,5678,xy,2015-06-03 21:00:00.000000 +0000,2015-06-04 12:34:56.000000 +0000,2015-06-04 23:45:06.000000 +0000,1970-01-01 06:04:02.000000 +0000,2015-06-04 01:02:03.123456 +0000
+10,,,,,,,,,,,,,2015-06-04 23:45:06.000000 +0000,,,
+11,99,9999,-99999999,-9999999999999999,1.2345000505447388,1.234567890123,-1234.0,1.2345678901234568E17,5678,xy,2015-06-03 21:00:00.000000 +0000,2015-06-04 12:34:56.000000 +0000,2015-06-04 23:45:06.000000 +0000,1970-01-01 06:04:02.000000 +0000,2015-06-04 01:02:03.123456 +0000,"{""key"":""value""}"

data/src/test/resources/org/embulk/input/mysql/test/expect/basic/test_long_expected.csv

@@ -1,2 +1,2 @@
-10,,,,,,,,,,,,,2015-06-04 23:45:06.000000 +0000,,
-11,99,9999,-99999999,-9999999999999999,1,1,-1234,123456789012345678,5678,,2015-06-03 21:00:00.000000 +0000,2015-06-04 12:34:56.000000 +0000,2015-06-04 23:45:06.000000 +0000,1970-01-01 06:04:02.000000 +0000,2015-06-04 01:02:03.123456 +0000
+10,,,,,,,,,,,,,2015-06-04 23:45:06.000000 +0000,,,
+11,99,9999,-99999999,-9999999999999999,1,1,-1234,123456789012345678,5678,,2015-06-03 21:00:00.000000 +0000,2015-06-04 12:34:56.000000 +0000,2015-06-04 23:45:06.000000 +0000,1970-01-01 06:04:02.000000 +0000,2015-06-04 01:02:03.123456 +0000,"{""key"":""value""}"

data/src/test/resources/org/embulk/input/mysql/test/expect/basic/test_string_expected.csv

@@ -1,2 +1,2 @@
-10,,,,,,,,,,,,,2015-06-04 23:45:06.000000 +0000,,
-11,99,9999,-99999999,-9999999999999999,1.2345,1.234567890123,-1234,123456789012345678.12,5678,xy,2015-06-03 21:00:00.000000 +0000,2015-06-04 12:34:56.000000 +0000,2015-06-04 23:45:06.000000 +0000,1970-01-01 06:04:02.000000 +0000,2015-06-04 01:02:03.123456 +0000
+10,,,,,,,,,,,,,2015-06-04 23:45:06.000000 +0000,,,
+11,99,9999,-99999999,-9999999999999999,1.2345,1.234567890123,-1234,123456789012345678.12,5678,xy,2015-06-03 21:00:00.000000 +0000,2015-06-04 12:34:56.000000 +0000,2015-06-04 23:45:06.000000 +0000,1970-01-01 06:04:02.000000 +0000,2015-06-04 01:02:03.123456 +0000,"{""key"":""value""}"

data/src/test/resources/org/embulk/input/mysql/test/expect/basic/test_timestamp1_expected.csv

@@ -1,2 +1,2 @@
-10,,,,,,,,,,,,,2015-06-04 23:45:06.000000 +0000,,
-11,99,9999,-99999999,-9999999999999999,1.2345000505447388,1.234567890123,-1234.0,1.2345678901234568E17,5678,xy,2015-06-03 21:00:00.000000 +0000,2015-06-04 12:34:56.000000 +0000,2015-06-04 23:45:06.000000 +0000,1970-01-01 06:04:02.000000 +0000,2015-06-04 01:02:03.123456 +0000
+10,,,,,,,,,,,,,2015-06-04 23:45:06.000000 +0000,,,
+11,99,9999,-99999999,-9999999999999999,1.2345000505447388,1.234567890123,-1234.0,1.2345678901234568E17,5678,xy,2015-06-03 21:00:00.000000 +0000,2015-06-04 12:34:56.000000 +0000,2015-06-04 23:45:06.000000 +0000,1970-01-01 06:04:02.000000 +0000,2015-06-04 01:02:03.123456 +0000,"{""key"":""value""}"

data/src/test/resources/org/embulk/input/mysql/test/expect/basic/test_timestamp2_expected.csv

@@ -1,2 +1,2 @@
-10,,,,,,,,,,,,,2015/06/05 08:45:06,,
-11,99,9999,-99999999,-9999999999999999,1.2345000505447388,1.234567890123,-1234.0,1.2345678901234568E17,5678,xy,2015/06/03,2015/06/04 12:34:56,2015/06/05 08:45:06,15-04-02,2015/06/04 01:02:03.123456
+10,,,,,,,,,,,,,2015/06/05 08:45:06,,,
+11,99,9999,-99999999,-9999999999999999,1.2345000505447388,1.234567890123,-1234.0,1.2345678901234568E17,5678,xy,2015/06/03,2015/06/04 12:34:56,2015/06/05 08:45:06,15-04-02,2015/06/04 01:02:03.123456,"{""key"":""value""}"

data/src/test/resources/org/embulk/input/mysql/test/expect/basic/test_timestamp3_expected.csv

@@ -1,2 +1,2 @@
-10,,,,,,,,,,,,,2015/06/05 08:45:06,,
-11,99,9999,-99999999,-9999999999999999,1.2345000505447388,1.234567890123,-1234.0,1.2345678901234568E17,5678,xy,2015/06/04,2015/06/04 21:34:56,2015/06/05 08:45:06,09-04-02,2015/06/04 10:02:03.123456
+10,,,,,,,,,,,,,2015/06/05 08:45:06,,,
+11,99,9999,-99999999,-9999999999999999,1.2345000505447388,1.234567890123,-1234.0,1.2345678901234568E17,5678,xy,2015/06/04,2015/06/04 21:34:56,2015/06/05 08:45:06,09-04-02,2015/06/04 10:02:03.123456,"{""key"":""value""}"

data/src/test/resources/org/embulk/input/mysql/test/expect/basic/test_valuetype_decimal_expected.csv

@@ -1,2 +1,2 @@
-10,,,,,,,,,,,,,2015-06-04 23:45:06.000000 +0000,,
-11,99.0,9999,-99999999,-9999999999999999,1.2345000505447388,1.234567890123,-1234.0,1.2345678901234568E17,5678,xy,2015-06-03 21:00:00.000000 +0000,2015-06-04 12:34:56.000000 +0000,2015-06-04 23:45:06.000000 +0000,1970-01-01 06:04:02.000000 +0000,2015-06-04 01:02:03.123456 +0000
+10,,,,,,,,,,,,,2015-06-04 23:45:06.000000 +0000,,,
+11,99.0,9999,-99999999,-9999999999999999,1.2345000505447388,1.234567890123,-1234.0,1.2345678901234568E17,5678,xy,2015-06-03 21:00:00.000000 +0000,2015-06-04 12:34:56.000000 +0000,2015-06-04 23:45:06.000000 +0000,1970-01-01 06:04:02.000000 +0000,2015-06-04 01:02:03.123456 +0000,"{""key"":""value""}"

data/src/test/resources/org/embulk/input/mysql/test/expect/basic/test_valuetype_json_config.yml

@@ -0,0 +1,3 @@
+table: test1
+column_options:
+  c10: {value_type: json}

data/src/test/resources/org/embulk/input/mysql/test/expect/basic/test_valuetype_json_expected.csv

@@ -0,0 +1,2 @@
+10,,,,,,,,,,,,,2015-06-04 23:45:06.000000 +0000,,,
+11,99,9999,-99999999,-9999999999999999,1.2345000505447388,1.234567890123,-1234.0,1.2345678901234568E17,5678,,2015-06-03 21:00:00.000000 +0000,2015-06-04 12:34:56.000000 +0000,2015-06-04 23:45:06.000000 +0000,1970-01-01 06:04:02.000000 +0000,2015-06-04 01:02:03.123456 +0000,"{""key"":""value""}"

data/src/test/resources/org/embulk/input/mysql/test/expect/basic/test_valuetype_json_expected.diff

@@ -0,0 +1,2 @@
+in: {}
+out: {}

data/src/test/resources/org/embulk/input/mysql/test/expect/basic/test_valuetype_string_config.yml

@@ -1,3 +1,4 @@
 table: test1
 column_options:
   c1: {value_type: string}
+  c16: {value_type: string}

data/src/test/resources/org/embulk/input/mysql/test/expect/basic/test_valuetype_string_expected.csv

@@ -1,2 +1,2 @@
-10,,,,,,,,,,,,,2015-06-04 23:45:06.000000 +0000,,
-11,99,9999,-99999999,-9999999999999999,1.2345000505447388,1.234567890123,-1234.0,1.2345678901234568E17,5678,xy,2015-06-03 21:00:00.000000 +0000,2015-06-04 12:34:56.000000 +0000,2015-06-04 23:45:06.000000 +0000,1970-01-01 06:04:02.000000 +0000,2015-06-04 01:02:03.123456 +0000
+10,,,,,,,,,,,,,2015-06-04 23:45:06.000000 +0000,,,
+11,99,9999,-99999999,-9999999999999999,1.2345000505447388,1.234567890123,-1234.0,1.2345678901234568E17,5678,xy,2015-06-03 21:00:00.000000 +0000,2015-06-04 12:34:56.000000 +0000,2015-06-04 23:45:06.000000 +0000,1970-01-01 06:04:02.000000 +0000,2015-06-04 01:02:03.123456 +0000,"{""key"":""value""}"

data/src/test/resources/org/embulk/input/mysql/test/expect/option/after_select.yml

@@ -0,0 +1,3 @@
+table: test1
+order_by: 'id'
+after_select: delete from test1 where id = 3

data/src/test/resources/org/embulk/input/mysql/test/expect/option/after_select_expected.csv

@@ -0,0 +1,3 @@
+1,A
+2,B
+4,D

data/src/test/resources/org/embulk/input/mysql/test/expect/option/before_select.yml

@@ -0,0 +1,3 @@
+table: test1
+order_by: 'id'
+before_select: delete from test1 where id = 3

data/src/test/resources/org/embulk/input/mysql/test/expect/option/before_select_expected.csv

@@ -0,0 +1,3 @@
+1,A
+2,B
+4,D

data/src/test/resources/org/embulk/input/mysql/test/expect/option/before_setup.yml

@@ -0,0 +1,6 @@
+table: test2
+order_by: 'id'
+before_setup: create table test2 (id int, num int); insert into test2 values(1, 999), (2, 0);
+d: insert into test2 values(1, 999), (2, 0);
+options:
+  allowMultiQueries: true

data/src/test/resources/org/embulk/input/mysql/test/expect/option/before_setup_expected.csv

@@ -0,0 +1,2 @@
+1,999
+2,0

data/src/test/resources/org/embulk/input/mysql/test/expect/option/column_options.yml

@@ -0,0 +1,5 @@
+table: test3
+order_by: 'id'
+column_options:
+  datetime1: {type: string, timestamp_format: '%Y/%m/%d %H:%M:%S', timezone: "+0000"}
+  datetime2: {type: string, timestamp_format: '%Y-%m-%d %H:%M:%S', timezone: "+0900"}

data/src/test/resources/org/embulk/input/mysql/test/expect/option/column_options_expected.csv

@@ -0,0 +1,3 @@
+1,,
+2,2019/01/02 12:34:56,2019-01-02 21:34:56
+3,2018/12/31 23:59:59,2019-01-01 08:59:59

data/src/test/resources/org/embulk/input/mysql/test/expect/option/default_column_options.yml

@@ -0,0 +1,6 @@
+table: test3
+order_by: 'id'
+default_column_options:
+  TIMESTAMP: {type: string, timestamp_format: '%Y/%m/%d %H:%M:%S', timezone: "+0000"}
+column_options:
+  datetime2: {type: string, timestamp_format: '%Y-%m-%d %H:%M:%S', timezone: "+0900"}

data/src/test/resources/org/embulk/input/mysql/test/expect/option/default_column_options_expected.csv

@@ -0,0 +1,3 @@
+1,,
+2,2019/01/02 12:34:56,2019-01-02 21:34:56
+3,2018/12/31 23:59:59,2019-01-01 08:59:59

data/src/test/resources/org/embulk/input/mysql/test/expect/option/expected1.csv

@@ -0,0 +1,4 @@
+1,A
+2,B
+3,C
+4,D

data/src/test/resources/org/embulk/input/mysql/test/expect/option/input.yml

@@ -0,0 +1,2 @@
+table: test1
+order_by: 'id'

data/src/test/resources/org/embulk/input/mysql/test/expect/option/setup.sql

@@ -0,0 +1,29 @@
+drop table if exists test1;
+
+create table test1 (
+    id   int not null,
+    name varchar(8) not null,
+    primary key(id)
+);
+
+insert into test1 values
+(1, 'A'),
+(2, 'B'),
+(3, 'C'),
+(4, 'D');
+
+drop table if exists test2;
+
+drop table if exists test3;
+
+create table test3 (
+    id         int not null,
+    datetime1  datetime,
+    datetime2  datetime,
+    primary key(id)
+);
+
+insert into test3 values
+(1, NULL, NULL),
+(2, '2019-01-02 12:34:56', '2019-01-02 12:34:56'),
+(3, '2018-12-31 23:59:59', '2018-12-31 23:59:59');

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: embulk-input-mysql
 version: !ruby/object:Gem::Version
-  version: 0.9.3
+  version: 0.10.0
 platform: ruby
 authors:
 - Sadayuki Furuhashi
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2018-08-10 00:00:00.000000000 Z
+date: 2019-05-23 00:00:00.000000000 Z
 dependencies: []
 description: Selects records from a table.
 email:
@@ -19,8 +19,8 @@ extra_rdoc_files: []
 files:
 - README.md
 - build.gradle
-- classpath/embulk-input-jdbc-0.9.3.jar
-- classpath/embulk-input-mysql-0.9.3.jar
+- classpath/embulk-input-jdbc-0.10.0.jar
+- classpath/embulk-input-mysql-0.10.0.jar
 - default_jdbc_driver/mysql-connector-java-5.1.44.jar
 - lib/embulk/input/mysql.rb
 - src/main/java/org/embulk/input/MySQLInputPlugin.java
@@ -31,7 +31,9 @@ files:
 - src/main/java/org/embulk/input/mysql/getter/MySQLTimestampTimestampIncrementalHandler.java
 - src/test/java/org/embulk/input/mysql/BasicTest.java
 - src/test/java/org/embulk/input/mysql/IncrementalTest.java
+- src/test/java/org/embulk/input/mysql/MySQLColumnGetterFactoryTest.java
 - src/test/java/org/embulk/input/mysql/MySQLTests.java
+- src/test/java/org/embulk/input/mysql/OptionTest.java
 - src/test/resources/org/embulk/input/mysql/test/expect/basic/setup.sql
 - src/test/resources/org/embulk/input/mysql/test/expect/basic/test_boolean_config.yml
 - src/test/resources/org/embulk/input/mysql/test/expect/basic/test_boolean_expected.csv
@@ -60,6 +62,9 @@ files:
 - src/test/resources/org/embulk/input/mysql/test/expect/basic/test_valuetype_decimal_config.yml
 - src/test/resources/org/embulk/input/mysql/test/expect/basic/test_valuetype_decimal_expected.csv
 - src/test/resources/org/embulk/input/mysql/test/expect/basic/test_valuetype_decimal_expected.diff
+- src/test/resources/org/embulk/input/mysql/test/expect/basic/test_valuetype_json_config.yml
+- src/test/resources/org/embulk/input/mysql/test/expect/basic/test_valuetype_json_expected.csv
+- src/test/resources/org/embulk/input/mysql/test/expect/basic/test_valuetype_json_expected.diff
 - src/test/resources/org/embulk/input/mysql/test/expect/basic/test_valuetype_string_config.yml
 - src/test/resources/org/embulk/input/mysql/test/expect/basic/test_valuetype_string_expected.csv
 - src/test/resources/org/embulk/input/mysql/test/expect/basic/test_valuetype_string_expected.diff
@@ -101,6 +106,19 @@ files:
 - src/test/resources/org/embulk/input/mysql/test/expect/incremental/ts/expected_2.diff
 - src/test/resources/org/embulk/input/mysql/test/expect/incremental/ts/insert_more.sql
 - src/test/resources/org/embulk/input/mysql/test/expect/incremental/ts/setup.sql
+- src/test/resources/org/embulk/input/mysql/test/expect/option/after_select.yml
+- src/test/resources/org/embulk/input/mysql/test/expect/option/after_select_expected.csv
+- src/test/resources/org/embulk/input/mysql/test/expect/option/before_select.yml
+- src/test/resources/org/embulk/input/mysql/test/expect/option/before_select_expected.csv
+- src/test/resources/org/embulk/input/mysql/test/expect/option/before_setup.yml
+- src/test/resources/org/embulk/input/mysql/test/expect/option/before_setup_expected.csv
+- src/test/resources/org/embulk/input/mysql/test/expect/option/column_options.yml
+- src/test/resources/org/embulk/input/mysql/test/expect/option/column_options_expected.csv
+- src/test/resources/org/embulk/input/mysql/test/expect/option/default_column_options.yml
+- src/test/resources/org/embulk/input/mysql/test/expect/option/default_column_options_expected.csv
+- src/test/resources/org/embulk/input/mysql/test/expect/option/expected1.csv
+- src/test/resources/org/embulk/input/mysql/test/expect/option/input.yml
+- src/test/resources/org/embulk/input/mysql/test/expect/option/setup.sql
 homepage: https://github.com/embulk/embulk-input-jdbc
 licenses:
 - Apache 2.0
@@ -121,7 +139,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project:
-rubygems_version: 2.4.8
+rubygems_version: 2.6.13
 signing_key:
 specification_version: 4
 summary: JDBC input plugin for Embulk