embulk-input-db2 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/README.md +148 -0
- data/build.gradle +7 -0
- data/classpath/embulk-input-db2-0.8.0.jar +0 -0
- data/classpath/embulk-input-jdbc-0.8.0.jar +0 -0
- data/lib/embulk/input/db2.rb +3 -0
- data/src/main/java/org/embulk/input/DB2InputPlugin.java +94 -0
- data/src/main/java/org/embulk/input/db2/DB2InputConnection.java +30 -0
- data/src/test/java/org/embulk/input/db2/DB2InputPluginTest.java +290 -0
- data/src/test/resources/db2/yml/input-datetime.yml +23 -0
- data/src/test/resources/db2/yml/input-number.yml +16 -0
- data/src/test/resources/db2/yml/input-string.yml +16 -0
- metadata +55 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA1:
|
|
3
|
+
metadata.gz: 22c93dbebbe181e15af9ee5200d4ff7883ebe074
|
|
4
|
+
data.tar.gz: 4f4636cbd2ba6fa6b05b1fcdacb82a96b5cd5fc6
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 2212f33180ac595f9ece95d990958c2a4644ee2a3e5ac27dfb6e3a379266cc7843733d0fff3c0f6769534041debfa127790bf729e475d96e196d2bd71f9d932d
|
|
7
|
+
data.tar.gz: f7396aa5648456ef879a18129494c5d9d4ddf77dcabf3692efeef242a4045afc81f223a99e03a745e09bffcd73d0f18ce631253fd3c6cd96ac4d181ad72252aa
|
data/README.md
ADDED
|
@@ -0,0 +1,148 @@
|
|
|
1
|
+
# DB2 input plugins for Embulk
|
|
2
|
+
|
|
3
|
+
DB2 input plugins for Embulk loads records from DB2.
|
|
4
|
+
|
|
5
|
+
## Overview
|
|
6
|
+
|
|
7
|
+
* **Plugin type**: input
|
|
8
|
+
* **Resume supported**: yes
|
|
9
|
+
|
|
10
|
+
## Configuration
|
|
11
|
+
|
|
12
|
+
- **driver_path**: path to the jar file of the DB2 JDBC driver (string)
|
|
13
|
+
- **host**: database host name (string, required)
|
|
14
|
+
- **port**: database port number (integer, default: 50000)
|
|
15
|
+
- **user**: database login user name (string, required)
|
|
16
|
+
- **password**: database login password (string)
|
|
17
|
+
- **database**: destination database name (string, required)
|
|
18
|
+
- **schema**: destination schema name (string, default: use the default schema)
|
|
19
|
+
- If you write SQL directly,
|
|
20
|
+
- **query**: SQL to run (string)
|
|
21
|
+
- If **query** is not set,
|
|
22
|
+
- **table**: destination table name (string, required)
|
|
23
|
+
- **select**: expression of select (e.g. `id, created_at`) (string, default: "*")
|
|
24
|
+
- **where**: WHERE condition to filter the rows (string, default: no-condition)
|
|
25
|
+
- **order_by**: expression of ORDER BY to sort rows (e.g. `created_at DESC, id ASC`) (string, default: not sorted)
|
|
26
|
+
- **fetch_rows**: number of rows to fetch one time (used for java.sql.Statement#setFetchSize) (integer, default: 10000)
|
|
27
|
+
- **connect_timeout**: timeout for the driver to connect. 0 means no timeout. (integer (seconds), default: 300)
|
|
28
|
+
- **socket_timeout**: timeout for executing the query. 0 means no timeout. (integer (seconds), default: 1800)
|
|
29
|
+
- **options**: extra JDBC properties (hash, default: {})
|
|
30
|
+
- **incremental**: if true, enables incremental loading. See next section for details (boolean, default: false)
|
|
31
|
+
- **incremental_columns**: column names for incremental loading (array of strings, default: use primary keys)
|
|
32
|
+
- **last_record**: values of the last record for incremental loading (array of objects, default: load all records)
|
|
33
|
+
- **default_timezone**: If the sql type of a column is `date`/`time`/`datetime` and the embulk type is `string`, column values are formatted int this default_timezone. You can overwrite timezone for each columns using column_options option. (string, default: `UTC`)
|
|
34
|
+
- **column_options**: advanced: a key-value pairs where key is a column name and value is options for the column.
|
|
35
|
+
- **value_type**: embulk get values from database as this value_type. Typically, the value_type determines `getXXX` method of `java.sql.PreparedStatement`.
|
|
36
|
+
(string, default: depends on the sql type of the column. Available values options are: `long`, `double`, `float`, `decimal`, `boolean`, `string`, `json`, `date`, `time`, `timestamp`)
|
|
37
|
+
- **type**: Column values are converted to this embulk type.
|
|
38
|
+
Available values options are: `boolean`, `long`, `double`, `string`, `json`, `timestamp`).
|
|
39
|
+
By default, the embulk type is determined according to the sql type of the column (or value_type if specified).
|
|
40
|
+
- **timestamp_format**: If the sql type of the column is `date`/`time`/`datetime` and the embulk type is `string`, column values are formatted by this timestamp_format. And if the embulk type is `timestamp`, this timestamp_format may be used in the output plugin. For example, stdout plugin use the timestamp_format, but *csv formatter plugin doesn't use*. (string, default : `%Y-%m-%d` for `date`, `%H:%M:%S` for `time`, `%Y-%m-%d %H:%M:%S` for `timestamp`)
|
|
41
|
+
- **timezone**: If the sql type of the column is `date`/`time`/`datetime` and the embulk type is `string`, column values are formatted in this timezone.
|
|
42
|
+
(string, value of default_timezone option is used by default)
|
|
43
|
+
- **after_select**: if set, this SQL will be executed after the SELECT query in the same transaction.
|
|
44
|
+
|
|
45
|
+
|
|
46
|
+
### Incremental loading
|
|
47
|
+
|
|
48
|
+
Incremental loading uses monotonically increasing unique columns (such as incremental (SEQUENCE) column) to load records inserted (or updated) after last execution.
|
|
49
|
+
|
|
50
|
+
First, if `incremental: true` is set, this plugin loads all records with additional ORDER BY. For example, if `incremental_columns: [updated_at, id]` option is set, query will be as following:
|
|
51
|
+
|
|
52
|
+
```
|
|
53
|
+
SELECT * FROM (
|
|
54
|
+
...original query is here...
|
|
55
|
+
)
|
|
56
|
+
ORDER BY updated_at, id
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
When bulk data loading finishes successfully, it outputs `last_record: ` paramater as config-diff so that next execution uses it.
|
|
60
|
+
|
|
61
|
+
At the next execution, when `last_record: ` is also set, this plugin generates additional WHERE conditions to load records larger than the last record. For example, if `last_record: ["2017-01-01 00:32:12", 5291]` is set,
|
|
62
|
+
|
|
63
|
+
```
|
|
64
|
+
SELECT * FROM (
|
|
65
|
+
...original query is here...
|
|
66
|
+
)
|
|
67
|
+
WHERE created_at > '2017-01-01 00:32:12' OR (created_at = '2017-01-01 00:32:12' AND id > 5291)
|
|
68
|
+
ORDER BY updated_at, id
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Then, it updates `last_record: ` so that next execution uses the updated last_record.
|
|
72
|
+
|
|
73
|
+
**IMPORTANT**: If you set `incremental_columns: ` option, make sure that there is an index on the columns to avoid full table scan. For this example, following index should be created:
|
|
74
|
+
|
|
75
|
+
```
|
|
76
|
+
CREATE INDEX embulk_incremental_loading_index ON table (updated_at, id);
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
Recommended usage is to leave `incremental_columns` unset and let this plugin automatically finds a primary key. Currently, only strings and integers are supported as incremental_columns.
|
|
80
|
+
|
|
81
|
+
|
|
82
|
+
## Example
|
|
83
|
+
|
|
84
|
+
```yaml
|
|
85
|
+
in:
|
|
86
|
+
type: db2
|
|
87
|
+
driver_path: /opt/db2/db2jcc4.jar
|
|
88
|
+
host: localhost
|
|
89
|
+
user: myuser
|
|
90
|
+
password: ""
|
|
91
|
+
database: my_database
|
|
92
|
+
table: my_table
|
|
93
|
+
select: "col1, col2, col3"
|
|
94
|
+
where: "col4 != 'a'"
|
|
95
|
+
order_by: "col1 DESC"
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
This configuration will generate following SQL:
|
|
99
|
+
|
|
100
|
+
```
|
|
101
|
+
SELECT col1, col2, col3
|
|
102
|
+
FROM "my_table"
|
|
103
|
+
WHERE col4 != 'a'
|
|
104
|
+
ORDER BY col1 DESC
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
If you need a complex SQL,
|
|
108
|
+
|
|
109
|
+
```yaml
|
|
110
|
+
in:
|
|
111
|
+
type: db2
|
|
112
|
+
driver_path: /opt/db2/db2jcc4.jar
|
|
113
|
+
host: localhost
|
|
114
|
+
user: myuser
|
|
115
|
+
password: ""
|
|
116
|
+
database: my_database
|
|
117
|
+
query: |
|
|
118
|
+
SELECT t1.id, t1.name, t2.id AS t2_id, t2.name AS t2_name
|
|
119
|
+
FROM table1 AS t1
|
|
120
|
+
LEFT JOIN table2 AS t2
|
|
121
|
+
ON t1.id = t2.t1_id
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
Advanced configuration:
|
|
125
|
+
|
|
126
|
+
```yaml
|
|
127
|
+
in:
|
|
128
|
+
type: db2
|
|
129
|
+
driver_path: /opt/db2/db2jcc4.jar
|
|
130
|
+
host: localhost
|
|
131
|
+
user: myuser
|
|
132
|
+
password: ""
|
|
133
|
+
database: my_database
|
|
134
|
+
table: "my_table"
|
|
135
|
+
select: "col1, col2, col3"
|
|
136
|
+
where: "col4 != 'a'"
|
|
137
|
+
column_options:
|
|
138
|
+
col1: {type: long}
|
|
139
|
+
col3: {type: string, timestamp_format: "%Y/%m/%d", timezone: "+0900"}
|
|
140
|
+
after_select: "update my_table set col5 = '1' where col4 != 'a'"
|
|
141
|
+
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
## Build
|
|
145
|
+
|
|
146
|
+
```
|
|
147
|
+
$ ./gradlew gem
|
|
148
|
+
```
|
data/build.gradle
ADDED
|
Binary file
|
|
Binary file
|
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
package org.embulk.input;
|
|
2
|
+
|
|
3
|
+
import java.sql.Connection;
|
|
4
|
+
import java.sql.DriverManager;
|
|
5
|
+
import java.sql.SQLException;
|
|
6
|
+
import java.util.Properties;
|
|
7
|
+
|
|
8
|
+
import org.embulk.config.Config;
|
|
9
|
+
import org.embulk.config.ConfigDefault;
|
|
10
|
+
import org.embulk.input.db2.DB2InputConnection;
|
|
11
|
+
import org.embulk.input.jdbc.AbstractJdbcInputPlugin;
|
|
12
|
+
import org.embulk.input.jdbc.JdbcInputConnection;
|
|
13
|
+
|
|
14
|
+
import com.google.common.base.Optional;
|
|
15
|
+
import static java.util.Locale.ENGLISH;
|
|
16
|
+
|
|
17
|
+
public class DB2InputPlugin
|
|
18
|
+
extends AbstractJdbcInputPlugin
|
|
19
|
+
{
|
|
20
|
+
public interface DB2PluginTask
|
|
21
|
+
extends PluginTask
|
|
22
|
+
{
|
|
23
|
+
@Config("driver_path")
|
|
24
|
+
@ConfigDefault("null")
|
|
25
|
+
public Optional<String> getDriverPath();
|
|
26
|
+
|
|
27
|
+
@Config("host")
|
|
28
|
+
public String getHost();
|
|
29
|
+
|
|
30
|
+
@Config("port")
|
|
31
|
+
@ConfigDefault("50000")
|
|
32
|
+
public int getPort();
|
|
33
|
+
|
|
34
|
+
@Config("database")
|
|
35
|
+
public String getDatabase();
|
|
36
|
+
|
|
37
|
+
@Config("schema")
|
|
38
|
+
@ConfigDefault("null")
|
|
39
|
+
public Optional<String> getSchema();
|
|
40
|
+
|
|
41
|
+
@Config("user")
|
|
42
|
+
public String getUser();
|
|
43
|
+
|
|
44
|
+
@Config("password")
|
|
45
|
+
@ConfigDefault("null")
|
|
46
|
+
public Optional<String> getPassword();
|
|
47
|
+
}
|
|
48
|
+
|
|
49
|
+
@Override
|
|
50
|
+
protected Class<? extends PluginTask> getTaskClass()
|
|
51
|
+
{
|
|
52
|
+
return DB2PluginTask.class;
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
@Override
|
|
56
|
+
protected JdbcInputConnection newConnection(PluginTask task) throws SQLException
|
|
57
|
+
{
|
|
58
|
+
DB2PluginTask db2Task = (DB2PluginTask) task;
|
|
59
|
+
|
|
60
|
+
String url = String.format(ENGLISH, "jdbc:db2://%s:%d/%s",
|
|
61
|
+
db2Task.getHost(), db2Task.getPort(), db2Task.getDatabase());
|
|
62
|
+
|
|
63
|
+
Properties props = new Properties();
|
|
64
|
+
props.setProperty("user", db2Task.getUser());
|
|
65
|
+
if (db2Task.getPassword().isPresent()) {
|
|
66
|
+
props.setProperty("password", db2Task.getPassword().get());
|
|
67
|
+
}
|
|
68
|
+
props.setProperty("connectionTimeout", String.valueOf(db2Task.getConnectTimeout() * 1000)); // milliseconds
|
|
69
|
+
props.setProperty("commandTimeout", String.valueOf(db2Task.getSocketTimeout() * 1000)); // milliseconds
|
|
70
|
+
props.putAll(db2Task.getOptions());
|
|
71
|
+
|
|
72
|
+
if (db2Task.getDriverPath().isPresent()) {
|
|
73
|
+
addDriverJarToClasspath(db2Task.getDriverPath().get());
|
|
74
|
+
}
|
|
75
|
+
|
|
76
|
+
try {
|
|
77
|
+
Class.forName("com.ibm.db2.jcc.DB2Driver");
|
|
78
|
+
} catch (Exception ex) {
|
|
79
|
+
throw new RuntimeException(ex);
|
|
80
|
+
}
|
|
81
|
+
|
|
82
|
+
Connection con = DriverManager.getConnection(url, props);
|
|
83
|
+
try {
|
|
84
|
+
DB2InputConnection c = new DB2InputConnection(con, db2Task.getSchema().orNull());
|
|
85
|
+
con = null;
|
|
86
|
+
return c;
|
|
87
|
+
} finally {
|
|
88
|
+
if (con != null) {
|
|
89
|
+
con.close();
|
|
90
|
+
}
|
|
91
|
+
}
|
|
92
|
+
}
|
|
93
|
+
|
|
94
|
+
}
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
package org.embulk.input.db2;
|
|
2
|
+
|
|
3
|
+
import java.sql.Connection;
|
|
4
|
+
import java.sql.SQLException;
|
|
5
|
+
|
|
6
|
+
import org.embulk.input.jdbc.JdbcInputConnection;
|
|
7
|
+
|
|
8
|
+
public class DB2InputConnection extends JdbcInputConnection {
|
|
9
|
+
|
|
10
|
+
public DB2InputConnection(Connection connection, String schemaName) throws SQLException
|
|
11
|
+
{
|
|
12
|
+
super(connection, schemaName);
|
|
13
|
+
}
|
|
14
|
+
|
|
15
|
+
@Override
|
|
16
|
+
protected void setSearchPath(String schema) throws SQLException
|
|
17
|
+
{
|
|
18
|
+
connection.setSchema(schema);
|
|
19
|
+
}
|
|
20
|
+
|
|
21
|
+
@Override
|
|
22
|
+
public void close() throws SQLException
|
|
23
|
+
{
|
|
24
|
+
// DB2 JDBC Driver requires explicit commit/rollback before closing connection.
|
|
25
|
+
connection.rollback();
|
|
26
|
+
|
|
27
|
+
super.close();
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
}
|
|
@@ -0,0 +1,290 @@
|
|
|
1
|
+
package org.embulk.input.db2;
|
|
2
|
+
|
|
3
|
+
import static java.util.Locale.ENGLISH;
|
|
4
|
+
import static org.junit.Assert.assertEquals;
|
|
5
|
+
|
|
6
|
+
import java.sql.Connection;
|
|
7
|
+
import java.sql.DriverManager;
|
|
8
|
+
import java.sql.SQLException;
|
|
9
|
+
import java.util.Arrays;
|
|
10
|
+
|
|
11
|
+
import org.embulk.input.AbstractJdbcInputPluginTest;
|
|
12
|
+
import org.embulk.input.DB2InputPlugin;
|
|
13
|
+
import org.embulk.spi.InputPlugin;
|
|
14
|
+
import org.junit.Test;
|
|
15
|
+
|
|
16
|
+
public class DB2InputPluginTest extends AbstractJdbcInputPluginTest
|
|
17
|
+
{
|
|
18
|
+
@Override
|
|
19
|
+
protected void prepare() throws SQLException
|
|
20
|
+
{
|
|
21
|
+
tester.addPlugin(InputPlugin.class, "db2", DB2InputPlugin.class);
|
|
22
|
+
|
|
23
|
+
try {
|
|
24
|
+
Class.forName("com.ibm.db2.jcc.DB2Driver");
|
|
25
|
+
} catch (ClassNotFoundException e) {
|
|
26
|
+
System.err.println("Warning: you should put 'db2jcc4.jar' in 'embulk-input-db2/driver' directory in order to test.");
|
|
27
|
+
return;
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
try {
|
|
31
|
+
connect();
|
|
32
|
+
} catch (SQLException e) {
|
|
33
|
+
System.err.println(e);
|
|
34
|
+
System.err.println(String.format(ENGLISH, "Warning: prepare a schema on DB2 (server = %s, port = %d, database = %s, user = %s, password = %s).",
|
|
35
|
+
getHost(), getPort(), getDatabase(), getUser(), getPassword()));
|
|
36
|
+
return;
|
|
37
|
+
}
|
|
38
|
+
|
|
39
|
+
enabled = true;
|
|
40
|
+
|
|
41
|
+
String dropString = "DROP TABLE TEST_STRING";
|
|
42
|
+
executeSQL(dropString, true);
|
|
43
|
+
|
|
44
|
+
String createString =
|
|
45
|
+
"CREATE TABLE TEST_STRING ("
|
|
46
|
+
+ "ID CHAR(2) NOT NULL,"
|
|
47
|
+
+ "CHAR_ITEM CHAR(4),"
|
|
48
|
+
+ "VARCHAR_ITEM VARCHAR(8),"
|
|
49
|
+
+ "CLOB_ITEM CLOB,"
|
|
50
|
+
+ "GRAPHIC_ITEM GRAPHIC(4),"
|
|
51
|
+
+ "VARGRAPHIC_ITEM VARGRAPHIC(8),"
|
|
52
|
+
+ "NCHAR_ITEM NCHAR(4),"
|
|
53
|
+
+ "NVARCHAR_ITEM NVARCHAR(8),"
|
|
54
|
+
+ "NCLOB_ITEM NCLOB,"
|
|
55
|
+
+ "PRIMARY KEY (ID))";
|
|
56
|
+
executeSQL(createString);
|
|
57
|
+
|
|
58
|
+
String insertString1 =
|
|
59
|
+
"INSERT INTO TEST_STRING VALUES("
|
|
60
|
+
+ "'10',"
|
|
61
|
+
+ "NULL,"
|
|
62
|
+
+ "NULL,"
|
|
63
|
+
+ "NULL,"
|
|
64
|
+
+ "NULL,"
|
|
65
|
+
+ "NULL,"
|
|
66
|
+
+ "NULL,"
|
|
67
|
+
+ "NULL,"
|
|
68
|
+
+ "NULL)";
|
|
69
|
+
executeSQL(insertString1);
|
|
70
|
+
|
|
71
|
+
String insertString2 =
|
|
72
|
+
"INSERT INTO TEST_STRING VALUES("
|
|
73
|
+
+ "'11',"
|
|
74
|
+
+ "'aa',"
|
|
75
|
+
+ "'AA',"
|
|
76
|
+
+ "'aaaaaaaaaaaa',"
|
|
77
|
+
+ "'ああ',"
|
|
78
|
+
+ "'いいいい',"
|
|
79
|
+
+ "'aa',"
|
|
80
|
+
+ "'AA',"
|
|
81
|
+
+ "'aaaaaaaa')";
|
|
82
|
+
executeSQL(insertString2);
|
|
83
|
+
|
|
84
|
+
String dropNumber = "DROP TABLE TEST_NUMBER";
|
|
85
|
+
executeSQL(dropNumber, true);
|
|
86
|
+
|
|
87
|
+
String createNumber =
|
|
88
|
+
"CREATE TABLE TEST_NUMBER ("
|
|
89
|
+
+ "ID CHAR(2) NOT NULL,"
|
|
90
|
+
+ "SMALLINT_ITEM SMALLINT,"
|
|
91
|
+
+ "INTEGER_ITEM INTEGER,"
|
|
92
|
+
+ "BIGINT_ITEM BIGINT,"
|
|
93
|
+
+ "DECIMAL_ITEM DECIMAL(8,2),"
|
|
94
|
+
+ "NUMERIC_ITEM NUMERIC(8,2),"
|
|
95
|
+
+ "REAL_ITEM REAL,"
|
|
96
|
+
+ "DOUBLE_ITEM DOUBLE,"
|
|
97
|
+
+ "FLOAT_ITEM FLOAT,"
|
|
98
|
+
+ "PRIMARY KEY (ID))";
|
|
99
|
+
executeSQL(createNumber);
|
|
100
|
+
|
|
101
|
+
String insertNumber1 =
|
|
102
|
+
"INSERT INTO TEST_NUMBER VALUES("
|
|
103
|
+
+ "'10',"
|
|
104
|
+
+ "NULL,"
|
|
105
|
+
+ "NULL,"
|
|
106
|
+
+ "NULL,"
|
|
107
|
+
+ "NULL,"
|
|
108
|
+
+ "NULL,"
|
|
109
|
+
+ "NULL,"
|
|
110
|
+
+ "NULL,"
|
|
111
|
+
+ "NULL)";
|
|
112
|
+
executeSQL(insertNumber1);
|
|
113
|
+
|
|
114
|
+
String insertNumber2 =
|
|
115
|
+
"INSERT INTO TEST_NUMBER VALUES("
|
|
116
|
+
+ "'11',"
|
|
117
|
+
+ "12345,"
|
|
118
|
+
+ "123456789,"
|
|
119
|
+
+ "123456789012,"
|
|
120
|
+
+ "123456.78,"
|
|
121
|
+
+ "876543.21,"
|
|
122
|
+
+ "1.23456,"
|
|
123
|
+
+ "1.23456789012,"
|
|
124
|
+
+ "3.45678901234)";
|
|
125
|
+
executeSQL(insertNumber2);
|
|
126
|
+
|
|
127
|
+
String insertNumber3 =
|
|
128
|
+
"INSERT INTO TEST_NUMBER VALUES("
|
|
129
|
+
+ "'12',"
|
|
130
|
+
+ "-12345,"
|
|
131
|
+
+ "-123456789,"
|
|
132
|
+
+ "-123456789012,"
|
|
133
|
+
+ "-123456.78,"
|
|
134
|
+
+ "-876543.21,"
|
|
135
|
+
+ "-1.23456,"
|
|
136
|
+
+ "-1.23456789012,"
|
|
137
|
+
+ "-3.45678901234)";
|
|
138
|
+
executeSQL(insertNumber3);
|
|
139
|
+
|
|
140
|
+
String dropDateTime = "DROP TABLE TEST_DATETIME";
|
|
141
|
+
executeSQL(dropDateTime, true);
|
|
142
|
+
|
|
143
|
+
String createDateTime =
|
|
144
|
+
"CREATE TABLE TEST_DATETIME ("
|
|
145
|
+
+ "ID CHAR(2) NOT NULL,"
|
|
146
|
+
+ "DATE_ITEM DATE,"
|
|
147
|
+
+ "TIME_ITEM TIME,"
|
|
148
|
+
+ "TIMESTAMP_ITEM TIMESTAMP,"
|
|
149
|
+
+ "TIMESTAMP0_ITEM TIMESTAMP(0),"
|
|
150
|
+
+ "TIMESTAMP12_ITEM TIMESTAMP(12),"
|
|
151
|
+
+ "PRIMARY KEY (ID))";
|
|
152
|
+
executeSQL(createDateTime);
|
|
153
|
+
|
|
154
|
+
String insertDateTime1 =
|
|
155
|
+
"INSERT INTO TEST_DATETIME VALUES("
|
|
156
|
+
+ "'10',"
|
|
157
|
+
+ "NULL,"
|
|
158
|
+
+ "NULL,"
|
|
159
|
+
+ "NULL,"
|
|
160
|
+
+ "NULL,"
|
|
161
|
+
+ "NULL)";
|
|
162
|
+
executeSQL(insertDateTime1);
|
|
163
|
+
|
|
164
|
+
String insertDateTime2 =
|
|
165
|
+
"INSERT INTO TEST_DATETIME VALUES("
|
|
166
|
+
+ "'11',"
|
|
167
|
+
+ "'2016-09-08',"
|
|
168
|
+
+ "'12:34:45',"
|
|
169
|
+
+ "'2016-09-09 12:34:45.123456',"
|
|
170
|
+
+ "'2016-09-10 12:34:45',"
|
|
171
|
+
+ "'2016-09-11 12:34:45.123456789012')";
|
|
172
|
+
executeSQL(insertDateTime2);
|
|
173
|
+
|
|
174
|
+
/*
|
|
175
|
+
String create1 =
|
|
176
|
+
"create table test1 ("
|
|
177
|
+
+ "c1 tinyint,"
|
|
178
|
+
+ "c2 smallint,"
|
|
179
|
+
+ "c3 int,"
|
|
180
|
+
+ "c4 bigint,"
|
|
181
|
+
+ "c5 float,"
|
|
182
|
+
+ "c6 double,"
|
|
183
|
+
+ "c7 decimal(4,0),"
|
|
184
|
+
+ "c8 decimal(20,2),"
|
|
185
|
+
+ "c9 char(4),"
|
|
186
|
+
+ "c10 varchar(4),"
|
|
187
|
+
+ "c11 date,"
|
|
188
|
+
+ "c12 datetime,"
|
|
189
|
+
+ "c13 timestamp,"
|
|
190
|
+
+ "c14 time,"
|
|
191
|
+
+ "c15 datetime(6));";
|
|
192
|
+
executeSQL(create1);
|
|
193
|
+
|
|
194
|
+
String insert1 =
|
|
195
|
+
"insert into test1 values("
|
|
196
|
+
+ "null,"
|
|
197
|
+
+ "null,"
|
|
198
|
+
+ "null,"
|
|
199
|
+
+ "null,"
|
|
200
|
+
+ "null,"
|
|
201
|
+
+ "null,"
|
|
202
|
+
+ "null,"
|
|
203
|
+
+ "null,"
|
|
204
|
+
+ "null,"
|
|
205
|
+
+ "null,"
|
|
206
|
+
+ "null,"
|
|
207
|
+
+ "null,"
|
|
208
|
+
+ "'2015-06-04 23:45:06',"
|
|
209
|
+
+ "null,"
|
|
210
|
+
+ "null);";
|
|
211
|
+
executeSQL(insert1);
|
|
212
|
+
|
|
213
|
+
String insert2 =
|
|
214
|
+
"insert into test1 values("
|
|
215
|
+
+ "99,"
|
|
216
|
+
+ "9999,"
|
|
217
|
+
+ "-99999999,"
|
|
218
|
+
+ "-9999999999999999,"
|
|
219
|
+
+ "1.2345,"
|
|
220
|
+
+ "1.234567890123,"
|
|
221
|
+
+ "-1234,"
|
|
222
|
+
+ "123456789012345678.12,"
|
|
223
|
+
+ "'5678',"
|
|
224
|
+
+ "'xy',"
|
|
225
|
+
+ "'2015-06-04',"
|
|
226
|
+
+ "'2015-06-04 12:34:56',"
|
|
227
|
+
+ "'2015-06-04 23:45:06',"
|
|
228
|
+
+ "'08:04:02',"
|
|
229
|
+
+ "'2015-06-04 01:02:03.123456');";
|
|
230
|
+
executeSQL(insert2);
|
|
231
|
+
|
|
232
|
+
String drop2 = "drop table if exists test2";
|
|
233
|
+
executeSQL(drop2);
|
|
234
|
+
|
|
235
|
+
String create2 = "create table test2 (c1 bigint unsigned);";
|
|
236
|
+
executeSQL(create2);
|
|
237
|
+
|
|
238
|
+
String insert3 = "insert into test2 values(18446744073709551615)";
|
|
239
|
+
executeSQL(insert3);*/
|
|
240
|
+
}
|
|
241
|
+
|
|
242
|
+
@Test
|
|
243
|
+
public void testString() throws Exception
|
|
244
|
+
{
|
|
245
|
+
if (enabled) {
|
|
246
|
+
test("/db2/yml/input-string.yml");
|
|
247
|
+
assertEquals(Arrays.asList(
|
|
248
|
+
"ID,CHAR_ITEM,VARCHAR_ITEM,CLOB_ITEM,GRAPHIC_ITEM,VARGRAPHIC_ITEM,NCHAR_ITEM,NVARCHAR_ITEM,NCLOB_ITEM",
|
|
249
|
+
"10,,,,,,,,",
|
|
250
|
+
"11,aa ,AA,aaaaaaaaaaaa,ああ ,いいいい,aa ,AA,aaaaaaaa"),
|
|
251
|
+
read("db2-input000.00.csv"));
|
|
252
|
+
}
|
|
253
|
+
}
|
|
254
|
+
|
|
255
|
+
@Test
|
|
256
|
+
public void testNumber() throws Exception
|
|
257
|
+
{
|
|
258
|
+
if (enabled) {
|
|
259
|
+
test("/db2/yml/input-number.yml");
|
|
260
|
+
assertEquals(Arrays.asList(
|
|
261
|
+
"ID,SMALLINT_ITEM,INTEGER_ITEM,BIGINT_ITEM,DECIMAL_ITEM,NUMERIC_ITEM,REAL_ITEM,DOUBLE_ITEM,FLOAT_ITEM",
|
|
262
|
+
"10,,,,,,,,",
|
|
263
|
+
// (double)1.23456f becomes "1.2345600128173828", not "1.23456", because of difference of precision.
|
|
264
|
+
"11,12345,123456789,123456789012,123456.78,876543.21," + (double)1.23456f + ",1.23456789012,3.45678901234",
|
|
265
|
+
"12,-12345,-123456789,-123456789012,-123456.78,-876543.21," + (double)-1.23456f + ",-1.23456789012,-3.45678901234"),
|
|
266
|
+
read("db2-input000.00.csv"));
|
|
267
|
+
}
|
|
268
|
+
}
|
|
269
|
+
|
|
270
|
+
@Test
|
|
271
|
+
public void testDateTime() throws Exception
|
|
272
|
+
{
|
|
273
|
+
if (enabled) {
|
|
274
|
+
test("/db2/yml/input-datetime.yml");
|
|
275
|
+
assertEquals(Arrays.asList(
|
|
276
|
+
"ID,DATE_ITEM,TIME_ITEM,TIMESTAMP_ITEM,TIMESTAMP0_ITEM,TIMESTAMP12_ITEM",
|
|
277
|
+
"10,,,,,",
|
|
278
|
+
// precision of embulk timestamp is nano seconds
|
|
279
|
+
"11,2016/09/07,03-34-45,2016/09/09 03:34:45,2016/09/10 12:34:45,2016/09/11 03:34:45.123456789000"),
|
|
280
|
+
read("db2-input000.00.csv"));
|
|
281
|
+
}
|
|
282
|
+
}
|
|
283
|
+
|
|
284
|
+
@Override
|
|
285
|
+
protected Connection connect() throws SQLException
|
|
286
|
+
{
|
|
287
|
+
return DriverManager.getConnection(String.format(ENGLISH, "jdbc:db2://%s:%d/%s", getHost(), getPort(), getDatabase()),
|
|
288
|
+
getUser(), getPassword());
|
|
289
|
+
}
|
|
290
|
+
}
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
in:
|
|
2
|
+
type: db2
|
|
3
|
+
host: #host#
|
|
4
|
+
database: #database#
|
|
5
|
+
user: #user#
|
|
6
|
+
password: #password#
|
|
7
|
+
table: TEST_DATETIME
|
|
8
|
+
select: "*"
|
|
9
|
+
order_by: "id"
|
|
10
|
+
default_timezone: '+0900'
|
|
11
|
+
|
|
12
|
+
out:
|
|
13
|
+
type: file
|
|
14
|
+
path_prefix: db2-input
|
|
15
|
+
file_ext: csv
|
|
16
|
+
formatter:
|
|
17
|
+
type: csv
|
|
18
|
+
column_options:
|
|
19
|
+
DATE_ITEM: {format: '%Y/%m/%d'}
|
|
20
|
+
TIME_ITEM: {format: '%H-%M-%S'}
|
|
21
|
+
TIMESTAMP_ITEM: {format: '%Y/%m/%d %H:%M:%S'}
|
|
22
|
+
TIMESTAMP0_ITEM: {format: '%Y/%m/%d %H:%M:%S', timezone: '+0900'}
|
|
23
|
+
TIMESTAMP12_ITEM: {format: '%Y/%m/%d %H:%M:%S.%12N'}
|
metadata
ADDED
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
|
2
|
+
name: embulk-input-db2
|
|
3
|
+
version: !ruby/object:Gem::Version
|
|
4
|
+
version: 0.8.0
|
|
5
|
+
platform: ruby
|
|
6
|
+
authors:
|
|
7
|
+
- Sadayuki Furuhashi
|
|
8
|
+
autorequire:
|
|
9
|
+
bindir: bin
|
|
10
|
+
cert_chain: []
|
|
11
|
+
date: 2016-10-13 00:00:00.000000000 Z
|
|
12
|
+
dependencies: []
|
|
13
|
+
description: Selects records from a table.
|
|
14
|
+
email:
|
|
15
|
+
- frsyuki@gmail.com
|
|
16
|
+
executables: []
|
|
17
|
+
extensions: []
|
|
18
|
+
extra_rdoc_files: []
|
|
19
|
+
files:
|
|
20
|
+
- README.md
|
|
21
|
+
- build.gradle
|
|
22
|
+
- classpath/embulk-input-db2-0.8.0.jar
|
|
23
|
+
- classpath/embulk-input-jdbc-0.8.0.jar
|
|
24
|
+
- lib/embulk/input/db2.rb
|
|
25
|
+
- src/main/java/org/embulk/input/DB2InputPlugin.java
|
|
26
|
+
- src/main/java/org/embulk/input/db2/DB2InputConnection.java
|
|
27
|
+
- src/test/java/org/embulk/input/db2/DB2InputPluginTest.java
|
|
28
|
+
- src/test/resources/db2/yml/input-datetime.yml
|
|
29
|
+
- src/test/resources/db2/yml/input-number.yml
|
|
30
|
+
- src/test/resources/db2/yml/input-string.yml
|
|
31
|
+
homepage: https://github.com/embulk/embulk-input-jdbc
|
|
32
|
+
licenses:
|
|
33
|
+
- Apache 2.0
|
|
34
|
+
metadata: {}
|
|
35
|
+
post_install_message:
|
|
36
|
+
rdoc_options: []
|
|
37
|
+
require_paths:
|
|
38
|
+
- lib
|
|
39
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
|
40
|
+
requirements:
|
|
41
|
+
- - ">="
|
|
42
|
+
- !ruby/object:Gem::Version
|
|
43
|
+
version: '0'
|
|
44
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
45
|
+
requirements:
|
|
46
|
+
- - ">="
|
|
47
|
+
- !ruby/object:Gem::Version
|
|
48
|
+
version: '0'
|
|
49
|
+
requirements: []
|
|
50
|
+
rubyforge_project:
|
|
51
|
+
rubygems_version: 2.4.8
|
|
52
|
+
signing_key:
|
|
53
|
+
specification_version: 4
|
|
54
|
+
summary: JDBC input plugin for Embulk
|
|
55
|
+
test_files: []
|