embulk-input-db2 0.8.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/README.md +148 -0
- data/build.gradle +7 -0
- data/classpath/embulk-input-db2-0.8.0.jar +0 -0
- data/classpath/embulk-input-jdbc-0.8.0.jar +0 -0
- data/lib/embulk/input/db2.rb +3 -0
- data/src/main/java/org/embulk/input/DB2InputPlugin.java +94 -0
- data/src/main/java/org/embulk/input/db2/DB2InputConnection.java +30 -0
- data/src/test/java/org/embulk/input/db2/DB2InputPluginTest.java +290 -0
- data/src/test/resources/db2/yml/input-datetime.yml +23 -0
- data/src/test/resources/db2/yml/input-number.yml +16 -0
- data/src/test/resources/db2/yml/input-string.yml +16 -0
- metadata +55 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 22c93dbebbe181e15af9ee5200d4ff7883ebe074
|
4
|
+
data.tar.gz: 4f4636cbd2ba6fa6b05b1fcdacb82a96b5cd5fc6
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 2212f33180ac595f9ece95d990958c2a4644ee2a3e5ac27dfb6e3a379266cc7843733d0fff3c0f6769534041debfa127790bf729e475d96e196d2bd71f9d932d
|
7
|
+
data.tar.gz: f7396aa5648456ef879a18129494c5d9d4ddf77dcabf3692efeef242a4045afc81f223a99e03a745e09bffcd73d0f18ce631253fd3c6cd96ac4d181ad72252aa
|
data/README.md
ADDED
@@ -0,0 +1,148 @@
|
|
1
|
+
# DB2 input plugins for Embulk
|
2
|
+
|
3
|
+
DB2 input plugins for Embulk loads records from DB2.
|
4
|
+
|
5
|
+
## Overview
|
6
|
+
|
7
|
+
* **Plugin type**: input
|
8
|
+
* **Resume supported**: yes
|
9
|
+
|
10
|
+
## Configuration
|
11
|
+
|
12
|
+
- **driver_path**: path to the jar file of the DB2 JDBC driver (string)
|
13
|
+
- **host**: database host name (string, required)
|
14
|
+
- **port**: database port number (integer, default: 50000)
|
15
|
+
- **user**: database login user name (string, required)
|
16
|
+
- **password**: database login password (string)
|
17
|
+
- **database**: destination database name (string, required)
|
18
|
+
- **schema**: destination schema name (string, default: use the default schema)
|
19
|
+
- If you write SQL directly,
|
20
|
+
- **query**: SQL to run (string)
|
21
|
+
- If **query** is not set,
|
22
|
+
- **table**: destination table name (string, required)
|
23
|
+
- **select**: expression of select (e.g. `id, created_at`) (string, default: "*")
|
24
|
+
- **where**: WHERE condition to filter the rows (string, default: no-condition)
|
25
|
+
- **order_by**: expression of ORDER BY to sort rows (e.g. `created_at DESC, id ASC`) (string, default: not sorted)
|
26
|
+
- **fetch_rows**: number of rows to fetch one time (used for java.sql.Statement#setFetchSize) (integer, default: 10000)
|
27
|
+
- **connect_timeout**: timeout for the driver to connect. 0 means no timeout. (integer (seconds), default: 300)
|
28
|
+
- **socket_timeout**: timeout for executing the query. 0 means no timeout. (integer (seconds), default: 1800)
|
29
|
+
- **options**: extra JDBC properties (hash, default: {})
|
30
|
+
- **incremental**: if true, enables incremental loading. See next section for details (boolean, default: false)
|
31
|
+
- **incremental_columns**: column names for incremental loading (array of strings, default: use primary keys)
|
32
|
+
- **last_record**: values of the last record for incremental loading (array of objects, default: load all records)
|
33
|
+
- **default_timezone**: If the sql type of a column is `date`/`time`/`datetime` and the embulk type is `string`, column values are formatted int this default_timezone. You can overwrite timezone for each columns using column_options option. (string, default: `UTC`)
|
34
|
+
- **column_options**: advanced: a key-value pairs where key is a column name and value is options for the column.
|
35
|
+
- **value_type**: embulk get values from database as this value_type. Typically, the value_type determines `getXXX` method of `java.sql.PreparedStatement`.
|
36
|
+
(string, default: depends on the sql type of the column. Available values options are: `long`, `double`, `float`, `decimal`, `boolean`, `string`, `json`, `date`, `time`, `timestamp`)
|
37
|
+
- **type**: Column values are converted to this embulk type.
|
38
|
+
Available values options are: `boolean`, `long`, `double`, `string`, `json`, `timestamp`).
|
39
|
+
By default, the embulk type is determined according to the sql type of the column (or value_type if specified).
|
40
|
+
- **timestamp_format**: If the sql type of the column is `date`/`time`/`datetime` and the embulk type is `string`, column values are formatted by this timestamp_format. And if the embulk type is `timestamp`, this timestamp_format may be used in the output plugin. For example, stdout plugin use the timestamp_format, but *csv formatter plugin doesn't use*. (string, default : `%Y-%m-%d` for `date`, `%H:%M:%S` for `time`, `%Y-%m-%d %H:%M:%S` for `timestamp`)
|
41
|
+
- **timezone**: If the sql type of the column is `date`/`time`/`datetime` and the embulk type is `string`, column values are formatted in this timezone.
|
42
|
+
(string, value of default_timezone option is used by default)
|
43
|
+
- **after_select**: if set, this SQL will be executed after the SELECT query in the same transaction.
|
44
|
+
|
45
|
+
|
46
|
+
### Incremental loading
|
47
|
+
|
48
|
+
Incremental loading uses monotonically increasing unique columns (such as incremental (SEQUENCE) column) to load records inserted (or updated) after last execution.
|
49
|
+
|
50
|
+
First, if `incremental: true` is set, this plugin loads all records with additional ORDER BY. For example, if `incremental_columns: [updated_at, id]` option is set, query will be as following:
|
51
|
+
|
52
|
+
```
|
53
|
+
SELECT * FROM (
|
54
|
+
...original query is here...
|
55
|
+
)
|
56
|
+
ORDER BY updated_at, id
|
57
|
+
```
|
58
|
+
|
59
|
+
When bulk data loading finishes successfully, it outputs `last_record: ` paramater as config-diff so that next execution uses it.
|
60
|
+
|
61
|
+
At the next execution, when `last_record: ` is also set, this plugin generates additional WHERE conditions to load records larger than the last record. For example, if `last_record: ["2017-01-01 00:32:12", 5291]` is set,
|
62
|
+
|
63
|
+
```
|
64
|
+
SELECT * FROM (
|
65
|
+
...original query is here...
|
66
|
+
)
|
67
|
+
WHERE created_at > '2017-01-01 00:32:12' OR (created_at = '2017-01-01 00:32:12' AND id > 5291)
|
68
|
+
ORDER BY updated_at, id
|
69
|
+
```
|
70
|
+
|
71
|
+
Then, it updates `last_record: ` so that next execution uses the updated last_record.
|
72
|
+
|
73
|
+
**IMPORTANT**: If you set `incremental_columns: ` option, make sure that there is an index on the columns to avoid full table scan. For this example, following index should be created:
|
74
|
+
|
75
|
+
```
|
76
|
+
CREATE INDEX embulk_incremental_loading_index ON table (updated_at, id);
|
77
|
+
```
|
78
|
+
|
79
|
+
Recommended usage is to leave `incremental_columns` unset and let this plugin automatically finds a primary key. Currently, only strings and integers are supported as incremental_columns.
|
80
|
+
|
81
|
+
|
82
|
+
## Example
|
83
|
+
|
84
|
+
```yaml
|
85
|
+
in:
|
86
|
+
type: db2
|
87
|
+
driver_path: /opt/db2/db2jcc4.jar
|
88
|
+
host: localhost
|
89
|
+
user: myuser
|
90
|
+
password: ""
|
91
|
+
database: my_database
|
92
|
+
table: my_table
|
93
|
+
select: "col1, col2, col3"
|
94
|
+
where: "col4 != 'a'"
|
95
|
+
order_by: "col1 DESC"
|
96
|
+
```
|
97
|
+
|
98
|
+
This configuration will generate following SQL:
|
99
|
+
|
100
|
+
```
|
101
|
+
SELECT col1, col2, col3
|
102
|
+
FROM "my_table"
|
103
|
+
WHERE col4 != 'a'
|
104
|
+
ORDER BY col1 DESC
|
105
|
+
```
|
106
|
+
|
107
|
+
If you need a complex SQL,
|
108
|
+
|
109
|
+
```yaml
|
110
|
+
in:
|
111
|
+
type: db2
|
112
|
+
driver_path: /opt/db2/db2jcc4.jar
|
113
|
+
host: localhost
|
114
|
+
user: myuser
|
115
|
+
password: ""
|
116
|
+
database: my_database
|
117
|
+
query: |
|
118
|
+
SELECT t1.id, t1.name, t2.id AS t2_id, t2.name AS t2_name
|
119
|
+
FROM table1 AS t1
|
120
|
+
LEFT JOIN table2 AS t2
|
121
|
+
ON t1.id = t2.t1_id
|
122
|
+
```
|
123
|
+
|
124
|
+
Advanced configuration:
|
125
|
+
|
126
|
+
```yaml
|
127
|
+
in:
|
128
|
+
type: db2
|
129
|
+
driver_path: /opt/db2/db2jcc4.jar
|
130
|
+
host: localhost
|
131
|
+
user: myuser
|
132
|
+
password: ""
|
133
|
+
database: my_database
|
134
|
+
table: "my_table"
|
135
|
+
select: "col1, col2, col3"
|
136
|
+
where: "col4 != 'a'"
|
137
|
+
column_options:
|
138
|
+
col1: {type: long}
|
139
|
+
col3: {type: string, timestamp_format: "%Y/%m/%d", timezone: "+0900"}
|
140
|
+
after_select: "update my_table set col5 = '1' where col4 != 'a'"
|
141
|
+
|
142
|
+
```
|
143
|
+
|
144
|
+
## Build
|
145
|
+
|
146
|
+
```
|
147
|
+
$ ./gradlew gem
|
148
|
+
```
|
data/build.gradle
ADDED
Binary file
|
Binary file
|
@@ -0,0 +1,94 @@
|
|
1
|
+
package org.embulk.input;
|
2
|
+
|
3
|
+
import java.sql.Connection;
|
4
|
+
import java.sql.DriverManager;
|
5
|
+
import java.sql.SQLException;
|
6
|
+
import java.util.Properties;
|
7
|
+
|
8
|
+
import org.embulk.config.Config;
|
9
|
+
import org.embulk.config.ConfigDefault;
|
10
|
+
import org.embulk.input.db2.DB2InputConnection;
|
11
|
+
import org.embulk.input.jdbc.AbstractJdbcInputPlugin;
|
12
|
+
import org.embulk.input.jdbc.JdbcInputConnection;
|
13
|
+
|
14
|
+
import com.google.common.base.Optional;
|
15
|
+
import static java.util.Locale.ENGLISH;
|
16
|
+
|
17
|
+
public class DB2InputPlugin
|
18
|
+
extends AbstractJdbcInputPlugin
|
19
|
+
{
|
20
|
+
public interface DB2PluginTask
|
21
|
+
extends PluginTask
|
22
|
+
{
|
23
|
+
@Config("driver_path")
|
24
|
+
@ConfigDefault("null")
|
25
|
+
public Optional<String> getDriverPath();
|
26
|
+
|
27
|
+
@Config("host")
|
28
|
+
public String getHost();
|
29
|
+
|
30
|
+
@Config("port")
|
31
|
+
@ConfigDefault("50000")
|
32
|
+
public int getPort();
|
33
|
+
|
34
|
+
@Config("database")
|
35
|
+
public String getDatabase();
|
36
|
+
|
37
|
+
@Config("schema")
|
38
|
+
@ConfigDefault("null")
|
39
|
+
public Optional<String> getSchema();
|
40
|
+
|
41
|
+
@Config("user")
|
42
|
+
public String getUser();
|
43
|
+
|
44
|
+
@Config("password")
|
45
|
+
@ConfigDefault("null")
|
46
|
+
public Optional<String> getPassword();
|
47
|
+
}
|
48
|
+
|
49
|
+
@Override
|
50
|
+
protected Class<? extends PluginTask> getTaskClass()
|
51
|
+
{
|
52
|
+
return DB2PluginTask.class;
|
53
|
+
}
|
54
|
+
|
55
|
+
@Override
|
56
|
+
protected JdbcInputConnection newConnection(PluginTask task) throws SQLException
|
57
|
+
{
|
58
|
+
DB2PluginTask db2Task = (DB2PluginTask) task;
|
59
|
+
|
60
|
+
String url = String.format(ENGLISH, "jdbc:db2://%s:%d/%s",
|
61
|
+
db2Task.getHost(), db2Task.getPort(), db2Task.getDatabase());
|
62
|
+
|
63
|
+
Properties props = new Properties();
|
64
|
+
props.setProperty("user", db2Task.getUser());
|
65
|
+
if (db2Task.getPassword().isPresent()) {
|
66
|
+
props.setProperty("password", db2Task.getPassword().get());
|
67
|
+
}
|
68
|
+
props.setProperty("connectionTimeout", String.valueOf(db2Task.getConnectTimeout() * 1000)); // milliseconds
|
69
|
+
props.setProperty("commandTimeout", String.valueOf(db2Task.getSocketTimeout() * 1000)); // milliseconds
|
70
|
+
props.putAll(db2Task.getOptions());
|
71
|
+
|
72
|
+
if (db2Task.getDriverPath().isPresent()) {
|
73
|
+
addDriverJarToClasspath(db2Task.getDriverPath().get());
|
74
|
+
}
|
75
|
+
|
76
|
+
try {
|
77
|
+
Class.forName("com.ibm.db2.jcc.DB2Driver");
|
78
|
+
} catch (Exception ex) {
|
79
|
+
throw new RuntimeException(ex);
|
80
|
+
}
|
81
|
+
|
82
|
+
Connection con = DriverManager.getConnection(url, props);
|
83
|
+
try {
|
84
|
+
DB2InputConnection c = new DB2InputConnection(con, db2Task.getSchema().orNull());
|
85
|
+
con = null;
|
86
|
+
return c;
|
87
|
+
} finally {
|
88
|
+
if (con != null) {
|
89
|
+
con.close();
|
90
|
+
}
|
91
|
+
}
|
92
|
+
}
|
93
|
+
|
94
|
+
}
|
@@ -0,0 +1,30 @@
|
|
1
|
+
package org.embulk.input.db2;
|
2
|
+
|
3
|
+
import java.sql.Connection;
|
4
|
+
import java.sql.SQLException;
|
5
|
+
|
6
|
+
import org.embulk.input.jdbc.JdbcInputConnection;
|
7
|
+
|
8
|
+
public class DB2InputConnection extends JdbcInputConnection {
|
9
|
+
|
10
|
+
public DB2InputConnection(Connection connection, String schemaName) throws SQLException
|
11
|
+
{
|
12
|
+
super(connection, schemaName);
|
13
|
+
}
|
14
|
+
|
15
|
+
@Override
|
16
|
+
protected void setSearchPath(String schema) throws SQLException
|
17
|
+
{
|
18
|
+
connection.setSchema(schema);
|
19
|
+
}
|
20
|
+
|
21
|
+
@Override
|
22
|
+
public void close() throws SQLException
|
23
|
+
{
|
24
|
+
// DB2 JDBC Driver requires explicit commit/rollback before closing connection.
|
25
|
+
connection.rollback();
|
26
|
+
|
27
|
+
super.close();
|
28
|
+
}
|
29
|
+
|
30
|
+
}
|
@@ -0,0 +1,290 @@
|
|
1
|
+
package org.embulk.input.db2;
|
2
|
+
|
3
|
+
import static java.util.Locale.ENGLISH;
|
4
|
+
import static org.junit.Assert.assertEquals;
|
5
|
+
|
6
|
+
import java.sql.Connection;
|
7
|
+
import java.sql.DriverManager;
|
8
|
+
import java.sql.SQLException;
|
9
|
+
import java.util.Arrays;
|
10
|
+
|
11
|
+
import org.embulk.input.AbstractJdbcInputPluginTest;
|
12
|
+
import org.embulk.input.DB2InputPlugin;
|
13
|
+
import org.embulk.spi.InputPlugin;
|
14
|
+
import org.junit.Test;
|
15
|
+
|
16
|
+
public class DB2InputPluginTest extends AbstractJdbcInputPluginTest
|
17
|
+
{
|
18
|
+
@Override
|
19
|
+
protected void prepare() throws SQLException
|
20
|
+
{
|
21
|
+
tester.addPlugin(InputPlugin.class, "db2", DB2InputPlugin.class);
|
22
|
+
|
23
|
+
try {
|
24
|
+
Class.forName("com.ibm.db2.jcc.DB2Driver");
|
25
|
+
} catch (ClassNotFoundException e) {
|
26
|
+
System.err.println("Warning: you should put 'db2jcc4.jar' in 'embulk-input-db2/driver' directory in order to test.");
|
27
|
+
return;
|
28
|
+
}
|
29
|
+
|
30
|
+
try {
|
31
|
+
connect();
|
32
|
+
} catch (SQLException e) {
|
33
|
+
System.err.println(e);
|
34
|
+
System.err.println(String.format(ENGLISH, "Warning: prepare a schema on DB2 (server = %s, port = %d, database = %s, user = %s, password = %s).",
|
35
|
+
getHost(), getPort(), getDatabase(), getUser(), getPassword()));
|
36
|
+
return;
|
37
|
+
}
|
38
|
+
|
39
|
+
enabled = true;
|
40
|
+
|
41
|
+
String dropString = "DROP TABLE TEST_STRING";
|
42
|
+
executeSQL(dropString, true);
|
43
|
+
|
44
|
+
String createString =
|
45
|
+
"CREATE TABLE TEST_STRING ("
|
46
|
+
+ "ID CHAR(2) NOT NULL,"
|
47
|
+
+ "CHAR_ITEM CHAR(4),"
|
48
|
+
+ "VARCHAR_ITEM VARCHAR(8),"
|
49
|
+
+ "CLOB_ITEM CLOB,"
|
50
|
+
+ "GRAPHIC_ITEM GRAPHIC(4),"
|
51
|
+
+ "VARGRAPHIC_ITEM VARGRAPHIC(8),"
|
52
|
+
+ "NCHAR_ITEM NCHAR(4),"
|
53
|
+
+ "NVARCHAR_ITEM NVARCHAR(8),"
|
54
|
+
+ "NCLOB_ITEM NCLOB,"
|
55
|
+
+ "PRIMARY KEY (ID))";
|
56
|
+
executeSQL(createString);
|
57
|
+
|
58
|
+
String insertString1 =
|
59
|
+
"INSERT INTO TEST_STRING VALUES("
|
60
|
+
+ "'10',"
|
61
|
+
+ "NULL,"
|
62
|
+
+ "NULL,"
|
63
|
+
+ "NULL,"
|
64
|
+
+ "NULL,"
|
65
|
+
+ "NULL,"
|
66
|
+
+ "NULL,"
|
67
|
+
+ "NULL,"
|
68
|
+
+ "NULL)";
|
69
|
+
executeSQL(insertString1);
|
70
|
+
|
71
|
+
String insertString2 =
|
72
|
+
"INSERT INTO TEST_STRING VALUES("
|
73
|
+
+ "'11',"
|
74
|
+
+ "'aa',"
|
75
|
+
+ "'AA',"
|
76
|
+
+ "'aaaaaaaaaaaa',"
|
77
|
+
+ "'ああ',"
|
78
|
+
+ "'いいいい',"
|
79
|
+
+ "'aa',"
|
80
|
+
+ "'AA',"
|
81
|
+
+ "'aaaaaaaa')";
|
82
|
+
executeSQL(insertString2);
|
83
|
+
|
84
|
+
String dropNumber = "DROP TABLE TEST_NUMBER";
|
85
|
+
executeSQL(dropNumber, true);
|
86
|
+
|
87
|
+
String createNumber =
|
88
|
+
"CREATE TABLE TEST_NUMBER ("
|
89
|
+
+ "ID CHAR(2) NOT NULL,"
|
90
|
+
+ "SMALLINT_ITEM SMALLINT,"
|
91
|
+
+ "INTEGER_ITEM INTEGER,"
|
92
|
+
+ "BIGINT_ITEM BIGINT,"
|
93
|
+
+ "DECIMAL_ITEM DECIMAL(8,2),"
|
94
|
+
+ "NUMERIC_ITEM NUMERIC(8,2),"
|
95
|
+
+ "REAL_ITEM REAL,"
|
96
|
+
+ "DOUBLE_ITEM DOUBLE,"
|
97
|
+
+ "FLOAT_ITEM FLOAT,"
|
98
|
+
+ "PRIMARY KEY (ID))";
|
99
|
+
executeSQL(createNumber);
|
100
|
+
|
101
|
+
String insertNumber1 =
|
102
|
+
"INSERT INTO TEST_NUMBER VALUES("
|
103
|
+
+ "'10',"
|
104
|
+
+ "NULL,"
|
105
|
+
+ "NULL,"
|
106
|
+
+ "NULL,"
|
107
|
+
+ "NULL,"
|
108
|
+
+ "NULL,"
|
109
|
+
+ "NULL,"
|
110
|
+
+ "NULL,"
|
111
|
+
+ "NULL)";
|
112
|
+
executeSQL(insertNumber1);
|
113
|
+
|
114
|
+
String insertNumber2 =
|
115
|
+
"INSERT INTO TEST_NUMBER VALUES("
|
116
|
+
+ "'11',"
|
117
|
+
+ "12345,"
|
118
|
+
+ "123456789,"
|
119
|
+
+ "123456789012,"
|
120
|
+
+ "123456.78,"
|
121
|
+
+ "876543.21,"
|
122
|
+
+ "1.23456,"
|
123
|
+
+ "1.23456789012,"
|
124
|
+
+ "3.45678901234)";
|
125
|
+
executeSQL(insertNumber2);
|
126
|
+
|
127
|
+
String insertNumber3 =
|
128
|
+
"INSERT INTO TEST_NUMBER VALUES("
|
129
|
+
+ "'12',"
|
130
|
+
+ "-12345,"
|
131
|
+
+ "-123456789,"
|
132
|
+
+ "-123456789012,"
|
133
|
+
+ "-123456.78,"
|
134
|
+
+ "-876543.21,"
|
135
|
+
+ "-1.23456,"
|
136
|
+
+ "-1.23456789012,"
|
137
|
+
+ "-3.45678901234)";
|
138
|
+
executeSQL(insertNumber3);
|
139
|
+
|
140
|
+
String dropDateTime = "DROP TABLE TEST_DATETIME";
|
141
|
+
executeSQL(dropDateTime, true);
|
142
|
+
|
143
|
+
String createDateTime =
|
144
|
+
"CREATE TABLE TEST_DATETIME ("
|
145
|
+
+ "ID CHAR(2) NOT NULL,"
|
146
|
+
+ "DATE_ITEM DATE,"
|
147
|
+
+ "TIME_ITEM TIME,"
|
148
|
+
+ "TIMESTAMP_ITEM TIMESTAMP,"
|
149
|
+
+ "TIMESTAMP0_ITEM TIMESTAMP(0),"
|
150
|
+
+ "TIMESTAMP12_ITEM TIMESTAMP(12),"
|
151
|
+
+ "PRIMARY KEY (ID))";
|
152
|
+
executeSQL(createDateTime);
|
153
|
+
|
154
|
+
String insertDateTime1 =
|
155
|
+
"INSERT INTO TEST_DATETIME VALUES("
|
156
|
+
+ "'10',"
|
157
|
+
+ "NULL,"
|
158
|
+
+ "NULL,"
|
159
|
+
+ "NULL,"
|
160
|
+
+ "NULL,"
|
161
|
+
+ "NULL)";
|
162
|
+
executeSQL(insertDateTime1);
|
163
|
+
|
164
|
+
String insertDateTime2 =
|
165
|
+
"INSERT INTO TEST_DATETIME VALUES("
|
166
|
+
+ "'11',"
|
167
|
+
+ "'2016-09-08',"
|
168
|
+
+ "'12:34:45',"
|
169
|
+
+ "'2016-09-09 12:34:45.123456',"
|
170
|
+
+ "'2016-09-10 12:34:45',"
|
171
|
+
+ "'2016-09-11 12:34:45.123456789012')";
|
172
|
+
executeSQL(insertDateTime2);
|
173
|
+
|
174
|
+
/*
|
175
|
+
String create1 =
|
176
|
+
"create table test1 ("
|
177
|
+
+ "c1 tinyint,"
|
178
|
+
+ "c2 smallint,"
|
179
|
+
+ "c3 int,"
|
180
|
+
+ "c4 bigint,"
|
181
|
+
+ "c5 float,"
|
182
|
+
+ "c6 double,"
|
183
|
+
+ "c7 decimal(4,0),"
|
184
|
+
+ "c8 decimal(20,2),"
|
185
|
+
+ "c9 char(4),"
|
186
|
+
+ "c10 varchar(4),"
|
187
|
+
+ "c11 date,"
|
188
|
+
+ "c12 datetime,"
|
189
|
+
+ "c13 timestamp,"
|
190
|
+
+ "c14 time,"
|
191
|
+
+ "c15 datetime(6));";
|
192
|
+
executeSQL(create1);
|
193
|
+
|
194
|
+
String insert1 =
|
195
|
+
"insert into test1 values("
|
196
|
+
+ "null,"
|
197
|
+
+ "null,"
|
198
|
+
+ "null,"
|
199
|
+
+ "null,"
|
200
|
+
+ "null,"
|
201
|
+
+ "null,"
|
202
|
+
+ "null,"
|
203
|
+
+ "null,"
|
204
|
+
+ "null,"
|
205
|
+
+ "null,"
|
206
|
+
+ "null,"
|
207
|
+
+ "null,"
|
208
|
+
+ "'2015-06-04 23:45:06',"
|
209
|
+
+ "null,"
|
210
|
+
+ "null);";
|
211
|
+
executeSQL(insert1);
|
212
|
+
|
213
|
+
String insert2 =
|
214
|
+
"insert into test1 values("
|
215
|
+
+ "99,"
|
216
|
+
+ "9999,"
|
217
|
+
+ "-99999999,"
|
218
|
+
+ "-9999999999999999,"
|
219
|
+
+ "1.2345,"
|
220
|
+
+ "1.234567890123,"
|
221
|
+
+ "-1234,"
|
222
|
+
+ "123456789012345678.12,"
|
223
|
+
+ "'5678',"
|
224
|
+
+ "'xy',"
|
225
|
+
+ "'2015-06-04',"
|
226
|
+
+ "'2015-06-04 12:34:56',"
|
227
|
+
+ "'2015-06-04 23:45:06',"
|
228
|
+
+ "'08:04:02',"
|
229
|
+
+ "'2015-06-04 01:02:03.123456');";
|
230
|
+
executeSQL(insert2);
|
231
|
+
|
232
|
+
String drop2 = "drop table if exists test2";
|
233
|
+
executeSQL(drop2);
|
234
|
+
|
235
|
+
String create2 = "create table test2 (c1 bigint unsigned);";
|
236
|
+
executeSQL(create2);
|
237
|
+
|
238
|
+
String insert3 = "insert into test2 values(18446744073709551615)";
|
239
|
+
executeSQL(insert3);*/
|
240
|
+
}
|
241
|
+
|
242
|
+
@Test
|
243
|
+
public void testString() throws Exception
|
244
|
+
{
|
245
|
+
if (enabled) {
|
246
|
+
test("/db2/yml/input-string.yml");
|
247
|
+
assertEquals(Arrays.asList(
|
248
|
+
"ID,CHAR_ITEM,VARCHAR_ITEM,CLOB_ITEM,GRAPHIC_ITEM,VARGRAPHIC_ITEM,NCHAR_ITEM,NVARCHAR_ITEM,NCLOB_ITEM",
|
249
|
+
"10,,,,,,,,",
|
250
|
+
"11,aa ,AA,aaaaaaaaaaaa,ああ ,いいいい,aa ,AA,aaaaaaaa"),
|
251
|
+
read("db2-input000.00.csv"));
|
252
|
+
}
|
253
|
+
}
|
254
|
+
|
255
|
+
@Test
|
256
|
+
public void testNumber() throws Exception
|
257
|
+
{
|
258
|
+
if (enabled) {
|
259
|
+
test("/db2/yml/input-number.yml");
|
260
|
+
assertEquals(Arrays.asList(
|
261
|
+
"ID,SMALLINT_ITEM,INTEGER_ITEM,BIGINT_ITEM,DECIMAL_ITEM,NUMERIC_ITEM,REAL_ITEM,DOUBLE_ITEM,FLOAT_ITEM",
|
262
|
+
"10,,,,,,,,",
|
263
|
+
// (double)1.23456f becomes "1.2345600128173828", not "1.23456", because of difference of precision.
|
264
|
+
"11,12345,123456789,123456789012,123456.78,876543.21," + (double)1.23456f + ",1.23456789012,3.45678901234",
|
265
|
+
"12,-12345,-123456789,-123456789012,-123456.78,-876543.21," + (double)-1.23456f + ",-1.23456789012,-3.45678901234"),
|
266
|
+
read("db2-input000.00.csv"));
|
267
|
+
}
|
268
|
+
}
|
269
|
+
|
270
|
+
@Test
|
271
|
+
public void testDateTime() throws Exception
|
272
|
+
{
|
273
|
+
if (enabled) {
|
274
|
+
test("/db2/yml/input-datetime.yml");
|
275
|
+
assertEquals(Arrays.asList(
|
276
|
+
"ID,DATE_ITEM,TIME_ITEM,TIMESTAMP_ITEM,TIMESTAMP0_ITEM,TIMESTAMP12_ITEM",
|
277
|
+
"10,,,,,",
|
278
|
+
// precision of embulk timestamp is nano seconds
|
279
|
+
"11,2016/09/07,03-34-45,2016/09/09 03:34:45,2016/09/10 12:34:45,2016/09/11 03:34:45.123456789000"),
|
280
|
+
read("db2-input000.00.csv"));
|
281
|
+
}
|
282
|
+
}
|
283
|
+
|
284
|
+
@Override
|
285
|
+
protected Connection connect() throws SQLException
|
286
|
+
{
|
287
|
+
return DriverManager.getConnection(String.format(ENGLISH, "jdbc:db2://%s:%d/%s", getHost(), getPort(), getDatabase()),
|
288
|
+
getUser(), getPassword());
|
289
|
+
}
|
290
|
+
}
|
@@ -0,0 +1,23 @@
|
|
1
|
+
in:
|
2
|
+
type: db2
|
3
|
+
host: #host#
|
4
|
+
database: #database#
|
5
|
+
user: #user#
|
6
|
+
password: #password#
|
7
|
+
table: TEST_DATETIME
|
8
|
+
select: "*"
|
9
|
+
order_by: "id"
|
10
|
+
default_timezone: '+0900'
|
11
|
+
|
12
|
+
out:
|
13
|
+
type: file
|
14
|
+
path_prefix: db2-input
|
15
|
+
file_ext: csv
|
16
|
+
formatter:
|
17
|
+
type: csv
|
18
|
+
column_options:
|
19
|
+
DATE_ITEM: {format: '%Y/%m/%d'}
|
20
|
+
TIME_ITEM: {format: '%H-%M-%S'}
|
21
|
+
TIMESTAMP_ITEM: {format: '%Y/%m/%d %H:%M:%S'}
|
22
|
+
TIMESTAMP0_ITEM: {format: '%Y/%m/%d %H:%M:%S', timezone: '+0900'}
|
23
|
+
TIMESTAMP12_ITEM: {format: '%Y/%m/%d %H:%M:%S.%12N'}
|
metadata
ADDED
@@ -0,0 +1,55 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: embulk-input-db2
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.8.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Sadayuki Furuhashi
|
8
|
+
autorequire:
|
9
|
+
bindir: bin
|
10
|
+
cert_chain: []
|
11
|
+
date: 2016-10-13 00:00:00.000000000 Z
|
12
|
+
dependencies: []
|
13
|
+
description: Selects records from a table.
|
14
|
+
email:
|
15
|
+
- frsyuki@gmail.com
|
16
|
+
executables: []
|
17
|
+
extensions: []
|
18
|
+
extra_rdoc_files: []
|
19
|
+
files:
|
20
|
+
- README.md
|
21
|
+
- build.gradle
|
22
|
+
- classpath/embulk-input-db2-0.8.0.jar
|
23
|
+
- classpath/embulk-input-jdbc-0.8.0.jar
|
24
|
+
- lib/embulk/input/db2.rb
|
25
|
+
- src/main/java/org/embulk/input/DB2InputPlugin.java
|
26
|
+
- src/main/java/org/embulk/input/db2/DB2InputConnection.java
|
27
|
+
- src/test/java/org/embulk/input/db2/DB2InputPluginTest.java
|
28
|
+
- src/test/resources/db2/yml/input-datetime.yml
|
29
|
+
- src/test/resources/db2/yml/input-number.yml
|
30
|
+
- src/test/resources/db2/yml/input-string.yml
|
31
|
+
homepage: https://github.com/embulk/embulk-input-jdbc
|
32
|
+
licenses:
|
33
|
+
- Apache 2.0
|
34
|
+
metadata: {}
|
35
|
+
post_install_message:
|
36
|
+
rdoc_options: []
|
37
|
+
require_paths:
|
38
|
+
- lib
|
39
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
40
|
+
requirements:
|
41
|
+
- - ">="
|
42
|
+
- !ruby/object:Gem::Version
|
43
|
+
version: '0'
|
44
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
45
|
+
requirements:
|
46
|
+
- - ">="
|
47
|
+
- !ruby/object:Gem::Version
|
48
|
+
version: '0'
|
49
|
+
requirements: []
|
50
|
+
rubyforge_project:
|
51
|
+
rubygems_version: 2.4.8
|
52
|
+
signing_key:
|
53
|
+
specification_version: 4
|
54
|
+
summary: JDBC input plugin for Embulk
|
55
|
+
test_files: []
|