embulk-filter-oracle_lookup 0.2.4 → 0.2.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +88 -73
- data/build.gradle +2 -2
- data/classpath/embulk-filter-oracle_lookup-0.2.6.jar +0 -0
- data/example/oracle-filter-lookup_testing.yml.liquid +40 -0
- data/example/oracle-filter-lookup_testing_csv.yml.liquid +53 -0
- data/example/oracle-filter-lookup_testing_mssql.yml.liquid +37 -0
- data/example/oracle-filter-lookup_testing_mysql.yml.liquid +34 -0
- data/example/oracle-filter-lookup_testing_postgres.yml.liquid +40 -0
- data/src/main/java/org/embulk/filter/oracle_lookup/OracleConnection.java +27 -51
- data/src/main/java/org/embulk/filter/oracle_lookup/OracleLookupFilterPlugin.java +38 -8
- metadata +8 -7
- data/calendar.csv +0 -10
- data/classpath/embulk-filter-oracle_lookup-0.2.4.jar +0 -0
- data/classpath/mysql-connector-java-8.0.19.jar +0 -0
- data/classpath/protobuf-java-3.6.1.jar +0 -0
- data/example/config.yml +0 -47
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 9ea3689641156187213e77675602cfd0dd5f9700d9856d959d35f1cac12377c9
|
|
4
|
+
data.tar.gz: 2dc91d6798a577c815e746e93600de7b43cbe78b198fc98e563587764c36ae24
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 0fb39c9e37315e0d434e687d435fef27f3d2fb4d13d0ec2297fbb854a74ebbb8052f3bd74724a30436613a36ed4df606f6e53d65dc400c3372d78e0269a302a8
|
|
7
|
+
data.tar.gz: 52f17dcf6010d4c5f146e651736245612bf5c2c0178007bf9a97dda784d0e4880a60570304868dd4c5ac184a5137b73040636b2cf7260378d219e06ed0fff5a2
|
data/README.md
CHANGED
|
@@ -5,22 +5,22 @@
|
|
|
5
5
|
<p align="center">Infoobjects is a consulting company that helps enterprises transform how and where they run applications and infrastructure.
|
|
6
6
|
From strategy, to implementation, to ongoing managed services, Infoobjects creates tailored cloud solutions for enterprises at all stages of the cloud journey.</p>
|
|
7
7
|
|
|
8
|
-
#
|
|
8
|
+
# Oracle lookup filter plugin for Embulk
|
|
9
9
|
[](LICENSE)
|
|
10
10
|
|
|
11
|
-
# Oracle lookup filter plugin for Embulk
|
|
12
11
|
|
|
13
12
|
An Embulk filter plugin for Lookup Transformation with Oracle database
|
|
14
13
|
|
|
15
14
|
## Configuration
|
|
16
15
|
|
|
17
16
|
- **oracle_lookup**: Required attributes for the LookUp Filter Plugin -
|
|
18
|
-
- **driver_path**: driver path for your driver which you installed in your system (example
|
|
19
|
-
- **driver_class**: driver class name (example `oracle.jdbc.driver.OracleDriver`)
|
|
20
|
-
- **host**: database host (example `localhost`)
|
|
21
|
-
- **port**: database port (example port for oracle `1521`) (required)
|
|
22
|
-
- **
|
|
23
|
-
- **
|
|
17
|
+
- **driver_path**: driver path for your driver which you installed in your system(required) (example `....Path..../jarname.jar`)
|
|
18
|
+
- **driver_class**: driver class name(required) (example `oracle.jdbc.driver.OracleDriver`)
|
|
19
|
+
- **host**: database host(required if url is not provided) (example `localhost`)
|
|
20
|
+
- **port**: database port(required if url is not provided) (example port for oracle `1521`) (required)
|
|
21
|
+
- **url**: url name (Either url or port,sid_name and host must be provided)
|
|
22
|
+
- **sid_name**: sid name (required if url is not provided)
|
|
23
|
+
- **table_name**: table name of your database (required)
|
|
24
24
|
- **username**: username for your database (required)
|
|
25
25
|
- **password**: password for database (required)
|
|
26
26
|
- **mapping_from**: (Name of columns to be matched with table 2 columns) (required)
|
|
@@ -30,93 +30,108 @@ An Embulk filter plugin for Lookup Transformation with Oracle database
|
|
|
30
30
|
- **Name of column-1**: column name-1 from input file
|
|
31
31
|
- **Name of column-2**: column name-2 from input file
|
|
32
32
|
- **new_columns**: (New generated column names) (required)
|
|
33
|
-
- **Name-1,Type-1**: Any Name, Type of the name (name:
|
|
34
|
-
- **Name-2,Type-2**: Any Name, Type of the name (name:
|
|
33
|
+
- **Name-1,Type-1**: Any Name, Type of the name (name: pin, type: string)
|
|
34
|
+
- **Name-2,Type-2**: Any Name, Type of the name (name: gender, type: string)
|
|
35
|
+
- **Name-3,Type-3**: Any Name, Type of the name (name: phone_number, type: string) etc ...
|
|
35
36
|
## Example - columns
|
|
36
37
|
|
|
37
|
-
|
|
38
|
-
|
|
38
|
+
Customer.csv for table 1 is as follows :-
|
|
39
|
+
|
|
39
40
|
```
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
41
|
+
id customer_name address email car_name company
|
|
42
|
+
1 John Doe 123 Main St, Anytown USA john.doe@example.com Civic Honda
|
|
43
|
+
2 Jane Smith 456 Elm St, Anytown USA jane.smith@example.com E-Class Mercedes-Benz
|
|
44
|
+
3 Bob Johnson 789 Oak St, Anytown USA bob.johnson@example.com GLE-Class Mercedes-Benz
|
|
45
|
+
4 Amanda Hernandez 999 Cedar St, Anytown USA amanda.hernandez@example.com 911 119
|
|
46
|
+
5 Tom Brown 567 Pine St, Anytown USA tom.brown@example.com C-Class Mercedes-Benz
|
|
47
|
+
6 Samantha Davis 890 Cedar St, Anytown USA samantha.davis@example.com Civic Honda
|
|
48
|
+
7 Mike Wilson 1234 Spruce St, Anytown USA mike.wilson@example.com GLE-Class Mercedes-Benz
|
|
49
|
+
8 Jason Brown 888 Pine St, Anytown USA jason.brown@example.com 911 Porsche
|
|
50
|
+
9 David Rodriguez 9010 Oak St, Anytown USA david.rodriguez@example.com GLC-Class Mercedes-Benz
|
|
51
|
+
10 Mark Davis 666 Spruce St, Anytown USA mark.davis@example.com C-Class Mercedes-Benz
|
|
52
|
+
11 Chris Thompson 222 Cedar St, Anytown USA chris.thompson@example.com Cayenne Porsche
|
|
53
|
+
12 Linda Young 555 Birch St, Anytown USA linda.young@example.com RAV4
|
|
54
|
+
13 Kevin Hernandez 444 Maple St, Anytown USA kevin.hernandez@example.com 911 119
|
|
50
55
|
```
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
56
|
+
|
|
57
|
+
Car.csv for table 2 is as follows :-
|
|
58
|
+
|
|
54
59
|
```
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
60
|
+
car_id model brand category fuel_capacity
|
|
61
|
+
87 GLE-Class Mercedes-Benz SUV 80
|
|
62
|
+
101 Cayenne Porsche SUV 75
|
|
63
|
+
119 911 Porsche Sports Car 64
|
|
64
|
+
205 Accord Honda Sedan 56
|
|
65
|
+
334 Pilot Honda SUV 70
|
|
66
|
+
434 CR-v Honda SUV 64
|
|
67
|
+
559 C-Class Mercedes-Benz Sedan 66
|
|
68
|
+
603 Civic Honda Sedan 42
|
|
69
|
+
697 E-Class Mercedes-Benz Sedan 72
|
|
70
|
+
812 GLC-Class Mercedes-Benz Sedan 68
|
|
71
|
+
|
|
72
|
+
|
|
68
73
|
```
|
|
69
|
-
|
|
74
|
+
|
|
70
75
|
As shown in yaml below, columns mentioned in mapping_from will be mapped with columns mentioned in mapping_to
|
|
71
76
|
ie:
|
|
72
77
|
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
country_name : country_address
|
|
78
|
+
car_name : model
|
|
79
|
+
company : brand
|
|
76
80
|
|
|
77
|
-
After successful mapping an Output.csv file containing the columns mentioned in new_columns will be generated
|
|
78
|
-
|
|
79
|
-
|
|
81
|
+
After successful mapping an Output.csv file containing the columns mentioned in new_columns will be generated
|
|
80
82
|
|
|
81
83
|
Output File generated :-
|
|
82
|
-
|
|
84
|
+
|
|
83
85
|
```
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
86
|
+
id customer_name address email car_name company car_id category fuel_capacity
|
|
87
|
+
1 John Doe 123 Main St, Anytown USA john.doe@example.com Civic Honda 603 Sedan 42
|
|
88
|
+
2 Jane Smith 456 Elm St, Anytown USA jane.smith@example.com E-Class Mercedes-Benz 697 Sedan 72
|
|
89
|
+
3 Bob Johnson 789 Oak St, Anytown USA bob.johnson@example.com GLE-Class Mercedes-Benz 87 SUV 80
|
|
90
|
+
4 Amanda Hernandez 999 Cedar St, Anytown USA amanda.hernandez@example.com 911 119 0
|
|
91
|
+
5 Tom Brown 567 Pine St, Anytown USA tom.brown@example.com C-Class Mercedes-Benz 559 Sedan 66
|
|
92
|
+
6 Samantha Davis 890 Cedar St, Anytown USA samantha.davis@example.com Civic Honda 603 Sedan 42
|
|
93
|
+
7 Mike Wilson 1234 Spruce St, Anytown USA mike.wilson@example.com GLE-Class Mercedes-Benz 87 SUV 80
|
|
94
|
+
8 Jason Brown 888 Pine St, Anytown USA jason.brown@example.com 911 Porsche 119 Sport Car 64
|
|
95
|
+
9 David Rodriguez 9010 Oak St, Anytown USA david.rodriguez@example.com GLC-Class Mercedes-Benz 812 SUV 68
|
|
96
|
+
10 Mark Davis 666 Spruce St, Anytown USA mark.davis@example.com C-Class Mercedes-Benz 559 Sedan 66
|
|
97
|
+
11 Chris Thompson 222 Cedar St, Anytown USA chris.thompson@example.com Cayenne Porsche 101 SUV 75
|
|
98
|
+
12 Linda Young 555 Birch St, Anytown USA linda.young@example.com RAV4 \N 0
|
|
99
|
+
13 Kevin Hernandez 444 Maple St, Anytown USA kevin.hernandez@example.com 911 119 0
|
|
100
|
+
|
|
94
101
|
```
|
|
95
102
|
|
|
96
103
|
|
|
97
104
|
|
|
98
105
|
```yaml
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
106
|
+
filters:
|
|
107
|
+
- type: oracle_lookup
|
|
108
|
+
url: jdbc:oracle:thin:@localhost:1521:orcl
|
|
109
|
+
driver_path: "C:/drivers/ojdbc8.jar"
|
|
110
|
+
driver_class: 'oracle.jdbc.driver.OracleDriver'
|
|
111
|
+
table_name: car
|
|
112
|
+
username: MYUSER
|
|
113
|
+
password: ABCD
|
|
114
|
+
mapping_from:
|
|
115
|
+
- car_name
|
|
116
|
+
- company
|
|
117
|
+
mapping_to:
|
|
118
|
+
- name_of_car
|
|
119
|
+
- brand
|
|
120
|
+
new_columns:
|
|
121
|
+
- { name: car_id, type: string }
|
|
122
|
+
- { name: category, type: string }
|
|
123
|
+
- { name: fuel_capacity, type: string }
|
|
115
124
|
```
|
|
116
125
|
|
|
117
126
|
Notes:
|
|
118
|
-
1. mapping_from attribute should be in same order as mentioned in input file.
|
|
127
|
+
1. mapping_from attribute should be in the same order as mentioned in the input file.
|
|
119
128
|
|
|
129
|
+
2. In case with JDBC plugin if any integer column returned as float/decimal then use to cast that column as long as below
|
|
130
|
+
```
|
|
131
|
+
column_options:
|
|
132
|
+
id: {value_type: long}
|
|
133
|
+
```
|
|
134
|
+
3. Matching columns data types must be int,long and String
|
|
120
135
|
## Development
|
|
121
136
|
|
|
122
137
|
Run example:
|
data/build.gradle
CHANGED
|
@@ -13,7 +13,7 @@ configurations {
|
|
|
13
13
|
provided
|
|
14
14
|
}
|
|
15
15
|
|
|
16
|
-
version = "0.2.
|
|
16
|
+
version = "0.2.6"
|
|
17
17
|
|
|
18
18
|
sourceCompatibility = 1.8
|
|
19
19
|
targetCompatibility = 1.8
|
|
@@ -23,7 +23,7 @@ dependencies {
|
|
|
23
23
|
provided "org.embulk:embulk-core:0.9.8"
|
|
24
24
|
// compile "YOUR_JAR_DEPENDENCY_GROUP:YOUR_JAR_DEPENDENCY_MODULE:YOUR_JAR_DEPENDENCY_VERSION"
|
|
25
25
|
testCompile "junit:junit:4.+"
|
|
26
|
-
|
|
26
|
+
implementation group: 'com.oracle.database.jdbc', name: 'ojdbc8', version: '21.3.0.0'
|
|
27
27
|
}
|
|
28
28
|
|
|
29
29
|
task classpath(type: Copy, dependsOn: ["jar"]) {
|
|
Binary file
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
in:
|
|
2
|
+
type: jdbc
|
|
3
|
+
driver_path: "C:/drivers/ojdbc8.jar"
|
|
4
|
+
driver_class: 'oracle.jdbc.driver.OracleDriver'
|
|
5
|
+
url: jdbc:oracle:thin:@localhost:1521:orcl
|
|
6
|
+
database: DEMO
|
|
7
|
+
user: MYUSER
|
|
8
|
+
password: ABCD
|
|
9
|
+
table: customers
|
|
10
|
+
column_options:
|
|
11
|
+
id: {value_type: long}
|
|
12
|
+
filters:
|
|
13
|
+
- type: oracle_lookup
|
|
14
|
+
url: jdbc:oracle:thin:@localhost:1521:orcl
|
|
15
|
+
driver_path: "C:/drivers/ojdbc8.jar"
|
|
16
|
+
driver_class: 'oracle.jdbc.driver.OracleDriver'
|
|
17
|
+
table_name: car
|
|
18
|
+
username: MYUSER
|
|
19
|
+
password: ABCD
|
|
20
|
+
mapping_from:
|
|
21
|
+
- car_name
|
|
22
|
+
- company
|
|
23
|
+
mapping_to:
|
|
24
|
+
- name_of_car
|
|
25
|
+
- brand
|
|
26
|
+
new_columns:
|
|
27
|
+
- { name: car_id, type: string }
|
|
28
|
+
- { name: category, type: string }
|
|
29
|
+
- { name: fuel_capacity, type: string }
|
|
30
|
+
|
|
31
|
+
out:
|
|
32
|
+
type: jdbc
|
|
33
|
+
driver_path: "C:/drivers/ojdbc8.jar"
|
|
34
|
+
driver_class: 'oracle.jdbc.driver.OracleDriver'
|
|
35
|
+
url: jdbc:oracle:thin:@localhost:1521:orcl
|
|
36
|
+
database: DEMO
|
|
37
|
+
user: MYUSER
|
|
38
|
+
password: ABCD
|
|
39
|
+
table: output
|
|
40
|
+
mode: insert_direct
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
exec:
|
|
2
|
+
max_threads: 1
|
|
3
|
+
min_output_tasks: 1
|
|
4
|
+
in:
|
|
5
|
+
type: file
|
|
6
|
+
path_prefix: "..path../customer.csv"
|
|
7
|
+
parser:
|
|
8
|
+
charset: UTF-8
|
|
9
|
+
type: csv
|
|
10
|
+
delimiter: ','
|
|
11
|
+
quote: '"'
|
|
12
|
+
header_line: true
|
|
13
|
+
columns:
|
|
14
|
+
- { name: id, type: long }
|
|
15
|
+
- { name: customer_name, type: string }
|
|
16
|
+
- { name: address, type: string }
|
|
17
|
+
- { name: email, type: string }
|
|
18
|
+
- { name: car_name, type: string }
|
|
19
|
+
- { name: company, type: string }
|
|
20
|
+
filters:
|
|
21
|
+
- type: oracle_lookup
|
|
22
|
+
url: jdbc:oracle:thin:@localhost:1521:orcl
|
|
23
|
+
driver_path: "C:/drivers/ojdbc8.jar"
|
|
24
|
+
driver_class: 'oracle.jdbc.driver.OracleDriver'
|
|
25
|
+
table_name: car
|
|
26
|
+
username: MYUSER
|
|
27
|
+
password: ABCD
|
|
28
|
+
mapping_from:
|
|
29
|
+
- car_name
|
|
30
|
+
- company
|
|
31
|
+
mapping_to:
|
|
32
|
+
- name_of_car
|
|
33
|
+
- brand
|
|
34
|
+
new_columns:
|
|
35
|
+
- { name: car_id, type: string }
|
|
36
|
+
- { name: category, type: string }
|
|
37
|
+
- { name: fuel_capacity, type: string }
|
|
38
|
+
|
|
39
|
+
out:
|
|
40
|
+
type: file
|
|
41
|
+
path_prefix: "..path../output.csv"
|
|
42
|
+
file_ext: csv
|
|
43
|
+
formatter:
|
|
44
|
+
type: csv
|
|
45
|
+
delimiter: ","
|
|
46
|
+
newline: CRLF
|
|
47
|
+
newline_in_field: LF
|
|
48
|
+
charset: UTF-8
|
|
49
|
+
quote_policy: MINIMAL
|
|
50
|
+
quote: '"'
|
|
51
|
+
escape: "\\"
|
|
52
|
+
null_string: "\\N"
|
|
53
|
+
default_timezone: 'UTC'
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
in:
|
|
2
|
+
type: sqlserver
|
|
3
|
+
host: Localhost
|
|
4
|
+
driver_path: "..path../mssql-jdbc-10.2.0.jre17.jar"
|
|
5
|
+
user: "usert"
|
|
6
|
+
password: "password"
|
|
7
|
+
database: "test"
|
|
8
|
+
table: customers
|
|
9
|
+
filters:
|
|
10
|
+
- type: oracle_lookup
|
|
11
|
+
url: jdbc:oracle:thin:@localhost:1521:orcl
|
|
12
|
+
driver_path: "C:/drivers/ojdbc8.jar"
|
|
13
|
+
driver_class: 'oracle.jdbc.driver.OracleDriver'
|
|
14
|
+
table_name: car
|
|
15
|
+
username: MYUSER
|
|
16
|
+
password: ABCD
|
|
17
|
+
mapping_from:
|
|
18
|
+
- car_name
|
|
19
|
+
- company
|
|
20
|
+
mapping_to:
|
|
21
|
+
- name_of_car
|
|
22
|
+
- brand
|
|
23
|
+
new_columns:
|
|
24
|
+
- { name: car_id, type: string }
|
|
25
|
+
- { name: category, type: string }
|
|
26
|
+
- { name: fuel_capacity, type: string }
|
|
27
|
+
|
|
28
|
+
|
|
29
|
+
out:
|
|
30
|
+
type: sqlserver
|
|
31
|
+
host: Localhost
|
|
32
|
+
driver_path: "..path../mssql-jdbc-10.2.0.jre17.jar"
|
|
33
|
+
user: "usert"
|
|
34
|
+
password: "password"
|
|
35
|
+
database: "test"
|
|
36
|
+
table: output_table
|
|
37
|
+
mode: truncate_insert
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
in:
|
|
2
|
+
type: mysql
|
|
3
|
+
host: localhost
|
|
4
|
+
user: root
|
|
5
|
+
password: 'passsword'
|
|
6
|
+
database: test
|
|
7
|
+
table: customers
|
|
8
|
+
filters:
|
|
9
|
+
- type: oracle_lookup
|
|
10
|
+
url: jdbc:oracle:thin:@localhost:1521:orcl
|
|
11
|
+
driver_path: "C:/drivers/ojdbc8.jar"
|
|
12
|
+
driver_class: 'oracle.jdbc.driver.OracleDriver'
|
|
13
|
+
table_name: car
|
|
14
|
+
username: MYUSER
|
|
15
|
+
password: ABCD
|
|
16
|
+
mapping_from:
|
|
17
|
+
- car_name
|
|
18
|
+
- company
|
|
19
|
+
mapping_to:
|
|
20
|
+
- name_of_car
|
|
21
|
+
- brand
|
|
22
|
+
new_columns:
|
|
23
|
+
- { name: car_id, type: string }
|
|
24
|
+
- { name: category, type: string }
|
|
25
|
+
- { name: fuel_capacity, type: string }
|
|
26
|
+
|
|
27
|
+
out:
|
|
28
|
+
type: mysql
|
|
29
|
+
host: localhost
|
|
30
|
+
user: root
|
|
31
|
+
password: 'passsword'
|
|
32
|
+
database: test
|
|
33
|
+
table: output_table
|
|
34
|
+
mode: truncate_insert
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
exec:
|
|
2
|
+
max_threads: 1
|
|
3
|
+
min_output_tasks: 1
|
|
4
|
+
in:
|
|
5
|
+
type: postgresql
|
|
6
|
+
host: localhost
|
|
7
|
+
port: 5432
|
|
8
|
+
user: postgres
|
|
9
|
+
password: 1234
|
|
10
|
+
schema: public
|
|
11
|
+
database: test
|
|
12
|
+
table: customers
|
|
13
|
+
filters:
|
|
14
|
+
- type: oracle_lookup
|
|
15
|
+
url: jdbc:oracle:thin:@localhost:1521:orcl
|
|
16
|
+
driver_path: "C:/drivers/ojdbc8.jar"
|
|
17
|
+
driver_class: 'oracle.jdbc.driver.OracleDriver'
|
|
18
|
+
table_name: car
|
|
19
|
+
username: MYUSER
|
|
20
|
+
password: ABCD
|
|
21
|
+
mapping_from:
|
|
22
|
+
- car_name
|
|
23
|
+
- company
|
|
24
|
+
mapping_to:
|
|
25
|
+
- name_of_car
|
|
26
|
+
- brand
|
|
27
|
+
new_columns:
|
|
28
|
+
- { name: car_id, type: string }
|
|
29
|
+
- { name: category, type: string }
|
|
30
|
+
- { name: fuel_capacity, type: string }
|
|
31
|
+
out:
|
|
32
|
+
type: postgresql
|
|
33
|
+
host: localhost
|
|
34
|
+
port: 5432
|
|
35
|
+
database: test
|
|
36
|
+
user: postgres
|
|
37
|
+
password: 1234
|
|
38
|
+
schema: public
|
|
39
|
+
table: output_table
|
|
40
|
+
mode: truncate_insert
|
|
@@ -25,12 +25,19 @@ public class OracleConnection {
|
|
|
25
25
|
|
|
26
26
|
private OracleConnection(OracleLookupFilterPlugin.PluginTask task) throws Exception {
|
|
27
27
|
try{
|
|
28
|
-
if(task.getDriverClass().isPresent()){
|
|
29
|
-
this.
|
|
28
|
+
if(task.getDriverClass().isPresent() && task.getDriverPath().isPresent()){
|
|
29
|
+
this.loadOracleJdbcDriver(task.getDriverClass().get(),task.getDriverPath());
|
|
30
30
|
}else{
|
|
31
|
-
this.
|
|
31
|
+
this.loadOracleJdbcDriver("oracle.jdbc.driver.OracleDriver",task.getDriverPath());
|
|
32
|
+
}
|
|
33
|
+
String url;
|
|
34
|
+
if(task.getURL().isPresent()){
|
|
35
|
+
url= task.getURL().get();
|
|
36
|
+
}else if(task.getSID().isPresent() && task.getHost().isPresent() && task.getPort().isPresent()){
|
|
37
|
+
url="jdbc:oracle:thin:@" + task.getHost().get() + ":"+task.getPort().get()+":"+task.getSID().get();
|
|
38
|
+
}else{
|
|
39
|
+
throw new RuntimeException("sid_name,host and port no must be provided by the user if url has not been provided");
|
|
32
40
|
}
|
|
33
|
-
String url = "jdbc:oracle:thin:@" + task.getHost() + ":"+task.getPort()+":"+task.getDatabase();
|
|
34
41
|
connection= DriverManager.getConnection(url, task.getUserName(), task.getPassword());
|
|
35
42
|
}catch (Exception e){
|
|
36
43
|
e.printStackTrace();
|
|
@@ -55,78 +62,47 @@ public class OracleConnection {
|
|
|
55
62
|
}
|
|
56
63
|
return connection;
|
|
57
64
|
}
|
|
58
|
-
private Class<? extends java.sql.Driver>
|
|
65
|
+
private Class<? extends java.sql.Driver> loadOracleJdbcDriver(
|
|
59
66
|
final String className,
|
|
60
67
|
final Optional<String> driverPath)
|
|
61
68
|
{
|
|
62
|
-
synchronized (
|
|
63
|
-
if (
|
|
64
|
-
return
|
|
69
|
+
synchronized (oracleJdbcDriver) {
|
|
70
|
+
if (oracleJdbcDriver.get() != null) {
|
|
71
|
+
return oracleJdbcDriver.get();
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
if (driverPath.isPresent()) {
|
|
75
|
+
logger.info(
|
|
76
|
+
"\"driver_path\" is set to load the Oracle JDBC driver class \"{}\". Adding it to classpath.", className);
|
|
77
|
+
this.addDriverJarToClasspath(driverPath.get());
|
|
65
78
|
}
|
|
66
79
|
|
|
67
80
|
try {
|
|
68
81
|
// If the class is found from the ClassLoader of the plugin, that is prioritized the highest.
|
|
69
|
-
final Class<? extends java.sql.Driver> found =
|
|
70
|
-
|
|
82
|
+
final Class<? extends java.sql.Driver> found = loadOracleJdbcDriverClassForName(className);
|
|
83
|
+
oracleJdbcDriver.compareAndSet(null, found);
|
|
71
84
|
|
|
72
85
|
if (driverPath.isPresent()) {
|
|
73
86
|
logger.warn(
|
|
74
|
-
"\"driver_path\" is set while the
|
|
87
|
+
"\"driver_path\" is set while the Oracle JDBC driver class \"{}\" is found from the PluginClassLoader."
|
|
75
88
|
+ " \"driver_path\" is ignored.", className);
|
|
76
89
|
}
|
|
77
90
|
return found;
|
|
78
91
|
}
|
|
79
92
|
catch (final ClassNotFoundException ex) {
|
|
80
|
-
|
|
81
|
-
}
|
|
82
|
-
|
|
83
|
-
if (driverPath.isPresent()) {
|
|
84
|
-
logger.info(
|
|
85
|
-
"\"driver_path\" is set to load the MySQL JDBC driver class \"{}\". Adding it to classpath.", className);
|
|
86
|
-
this.addDriverJarToClasspath(driverPath.get());
|
|
87
|
-
}
|
|
88
|
-
else {
|
|
89
|
-
final File root = this.findPluginRoot();
|
|
90
|
-
final File driverLib = new File(root, "default_jdbc_driver");
|
|
91
|
-
final File[] files = driverLib.listFiles(new FileFilter() {
|
|
92
|
-
@Override
|
|
93
|
-
public boolean accept(final File file)
|
|
94
|
-
{
|
|
95
|
-
return file.isFile() && file.getName().endsWith(".jar");
|
|
96
|
-
}
|
|
97
|
-
});
|
|
98
|
-
if (files == null || files.length == 0) {
|
|
99
|
-
throw new ConfigException(new ClassNotFoundException(
|
|
100
|
-
"The MySQL JDBC driver for the class \"" + className + "\" is not found"
|
|
101
|
-
+ " in \"default_jdbc_driver\" (" + root.getAbsolutePath() + ")."));
|
|
102
|
-
}
|
|
103
|
-
for (final File file : files) {
|
|
104
|
-
logger.info(
|
|
105
|
-
"The MySQL JDBC driver for the class \"{}\" is expected to be found"
|
|
106
|
-
+ " in \"default_jdbc_driver\" at {}.", className, file.getAbsolutePath());
|
|
107
|
-
this.addDriverJarToClasspath(file.getAbsolutePath());
|
|
108
|
-
}
|
|
93
|
+
throw new ConfigException("The Oracle JDBC driver for the class \"" + className + "\" is not found.", ex);
|
|
109
94
|
}
|
|
110
95
|
|
|
111
|
-
try {
|
|
112
|
-
// Retrying to find the class from the ClassLoader of the plugin.
|
|
113
|
-
final Class<? extends java.sql.Driver> found = loadJdbcDriverClassForName(className);
|
|
114
|
-
mysqlJdbcDriver.compareAndSet(null, found);
|
|
115
|
-
return found;
|
|
116
|
-
}
|
|
117
|
-
catch (final ClassNotFoundException ex) {
|
|
118
|
-
throw new ConfigException("The MySQL JDBC driver for the class \"" + className + "\" is not found.", ex);
|
|
119
|
-
}
|
|
120
96
|
}
|
|
121
97
|
}
|
|
122
98
|
|
|
123
99
|
@SuppressWarnings("unchecked")
|
|
124
|
-
private static Class<? extends java.sql.Driver>
|
|
100
|
+
private static Class<? extends java.sql.Driver> loadOracleJdbcDriverClassForName(final String className) throws ClassNotFoundException
|
|
125
101
|
{
|
|
126
102
|
return (Class<? extends java.sql.Driver>) Class.forName(className);
|
|
127
103
|
}
|
|
128
104
|
|
|
129
|
-
private static final AtomicReference<Class<? extends java.sql.Driver>>
|
|
105
|
+
private static final AtomicReference<Class<? extends java.sql.Driver>> oracleJdbcDriver = new AtomicReference<>();
|
|
130
106
|
|
|
131
107
|
private static final Logger logger = LoggerFactory.getLogger(OracleConnection.class);
|
|
132
108
|
|
|
@@ -5,25 +5,29 @@ import org.embulk.config.*;
|
|
|
5
5
|
import org.embulk.spi.*;
|
|
6
6
|
import org.embulk.spi.time.Timestamp;
|
|
7
7
|
import org.embulk.spi.type.Types;
|
|
8
|
+
import org.slf4j.Logger;
|
|
9
|
+
import org.slf4j.LoggerFactory;
|
|
10
|
+
|
|
8
11
|
import java.sql.*;
|
|
9
12
|
import java.time.Instant;
|
|
10
13
|
import java.util.*;
|
|
11
14
|
|
|
12
15
|
public class OracleLookupFilterPlugin
|
|
13
16
|
implements FilterPlugin {
|
|
17
|
+
private static final Logger logger = LoggerFactory.getLogger(OracleLookupFilterPlugin.class);
|
|
18
|
+
|
|
14
19
|
public interface PluginTask
|
|
15
20
|
extends Task {
|
|
16
21
|
|
|
17
22
|
@Config("host")
|
|
18
|
-
|
|
23
|
+
@ConfigDefault("null")
|
|
24
|
+
public Optional<String> getHost();
|
|
19
25
|
|
|
20
26
|
@Config("port")
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
@Config("database")
|
|
24
|
-
public String getDatabase();
|
|
27
|
+
@ConfigDefault("null")
|
|
28
|
+
public Optional<String> getPort();
|
|
25
29
|
|
|
26
|
-
@Config("
|
|
30
|
+
@Config("table_name")
|
|
27
31
|
public String getTableName();
|
|
28
32
|
|
|
29
33
|
@Config("username")
|
|
@@ -48,6 +52,14 @@ public class OracleLookupFilterPlugin
|
|
|
48
52
|
@Config("driver_class")
|
|
49
53
|
@ConfigDefault("null")
|
|
50
54
|
public Optional<String> getDriverClass();
|
|
55
|
+
|
|
56
|
+
@Config("sid_name")
|
|
57
|
+
@ConfigDefault("null")
|
|
58
|
+
public Optional<String> getSID();
|
|
59
|
+
|
|
60
|
+
@Config("url")
|
|
61
|
+
@ConfigDefault("null")
|
|
62
|
+
public Optional<String> getURL();
|
|
51
63
|
}
|
|
52
64
|
|
|
53
65
|
|
|
@@ -132,7 +144,7 @@ public class OracleLookupFilterPlugin
|
|
|
132
144
|
columnNeedsToBeFetched += newColumns.get(i);
|
|
133
145
|
}
|
|
134
146
|
}
|
|
135
|
-
query += columnNeedsToBeFetched + " from "+task.
|
|
147
|
+
query += columnNeedsToBeFetched + " from "+task.getTableName();
|
|
136
148
|
|
|
137
149
|
Statement stmt = con.createStatement();
|
|
138
150
|
ResultSet rs = stmt.executeQuery(query);
|
|
@@ -200,6 +212,9 @@ public class OracleLookupFilterPlugin
|
|
|
200
212
|
columnConfigList.add(columnConfig);
|
|
201
213
|
}
|
|
202
214
|
|
|
215
|
+
Set<String> unmatchedData = new LinkedHashSet<>();
|
|
216
|
+
List<String> keyColumns = task.getMappingFrom();
|
|
217
|
+
|
|
203
218
|
while (reader.nextRecord()) {
|
|
204
219
|
|
|
205
220
|
int colNum = 0;
|
|
@@ -232,6 +247,8 @@ public class OracleLookupFilterPlugin
|
|
|
232
247
|
List<String> matchedData = new ArrayList<>();
|
|
233
248
|
if (keyValuePair.containsKey(key)) {
|
|
234
249
|
matchedData = keyValuePair.get(key);
|
|
250
|
+
}else{
|
|
251
|
+
unmatchedData.add(key);
|
|
235
252
|
}
|
|
236
253
|
|
|
237
254
|
if (matchedData.size() == 0) {
|
|
@@ -247,6 +264,18 @@ public class OracleLookupFilterPlugin
|
|
|
247
264
|
}
|
|
248
265
|
builder.addRecord();
|
|
249
266
|
}
|
|
267
|
+
String info="\n--------------------Unmatched rows.....................\nMapping Key Columns: ";
|
|
268
|
+
for(int i=0;i<keyColumns.size();i++){
|
|
269
|
+
info+= keyColumns.get(i);
|
|
270
|
+
if(i!=keyColumns.size()-1){
|
|
271
|
+
info+=",";
|
|
272
|
+
}
|
|
273
|
+
}
|
|
274
|
+
info+="\n";
|
|
275
|
+
for(String key: unmatchedData){
|
|
276
|
+
info+= key+"\n";
|
|
277
|
+
}
|
|
278
|
+
logger.info(info);
|
|
250
279
|
}
|
|
251
280
|
|
|
252
281
|
|
|
@@ -282,7 +311,8 @@ public class OracleLookupFilterPlugin
|
|
|
282
311
|
} else if (Types.DOUBLE.equals(column.getType())) {
|
|
283
312
|
if (keyMap.get("Key") < inputColumns.size()) {
|
|
284
313
|
if (column.getName().equalsIgnoreCase(inputColumns.get(keyMap.get("Key")))) {
|
|
285
|
-
|
|
314
|
+
long num = (long) reader.getDouble(column);
|
|
315
|
+
searchingKeyData.add(String.valueOf(num));
|
|
286
316
|
int key = keyMap.get("Key");
|
|
287
317
|
keyMap.put("Key", ++key);
|
|
288
318
|
}
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: embulk-filter-oracle_lookup
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.2.
|
|
4
|
+
version: 0.2.6
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Infoobjects Inc.
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2023-
|
|
11
|
+
date: 2023-03-03 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: bundler
|
|
@@ -49,13 +49,14 @@ files:
|
|
|
49
49
|
- LICENSE
|
|
50
50
|
- README.md
|
|
51
51
|
- build.gradle
|
|
52
|
-
-
|
|
53
|
-
- classpath/embulk-filter-oracle_lookup-0.2.4.jar
|
|
54
|
-
- classpath/mysql-connector-java-8.0.19.jar
|
|
55
|
-
- classpath/protobuf-java-3.6.1.jar
|
|
52
|
+
- classpath/embulk-filter-oracle_lookup-0.2.6.jar
|
|
56
53
|
- config/checkstyle/checkstyle.xml
|
|
57
54
|
- config/checkstyle/default.xml
|
|
58
|
-
- example/
|
|
55
|
+
- example/oracle-filter-lookup_testing.yml.liquid
|
|
56
|
+
- example/oracle-filter-lookup_testing_csv.yml.liquid
|
|
57
|
+
- example/oracle-filter-lookup_testing_mssql.yml.liquid
|
|
58
|
+
- example/oracle-filter-lookup_testing_mysql.yml.liquid
|
|
59
|
+
- example/oracle-filter-lookup_testing_postgres.yml.liquid
|
|
59
60
|
- gradle/wrapper/gradle-wrapper.jar
|
|
60
61
|
- gradle/wrapper/gradle-wrapper.properties
|
|
61
62
|
- gradlew
|
data/calendar.csv
DELETED
|
@@ -1,10 +0,0 @@
|
|
|
1
|
-
dim_calendar_key,year_number,quarter_number,attr_1
|
|
2
|
-
-1,0,0
|
|
3
|
-
19900101,1990,1,FirstIndia
|
|
4
|
-
19900102,1990,2,SecondUSA
|
|
5
|
-
19900103,1990,,1000.56
|
|
6
|
-
19900104,1990,4,40_China
|
|
7
|
-
19900105,1990,5,50_Ukraine
|
|
8
|
-
19900106,1990,2,10_USA
|
|
9
|
-
19900107,1990,3,30_UK
|
|
10
|
-
19900109,1990,7,
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
data/example/config.yml
DELETED
|
@@ -1,47 +0,0 @@
|
|
|
1
|
-
exec:
|
|
2
|
-
max_threads: 2
|
|
3
|
-
min_output_tasks: 1
|
|
4
|
-
in:
|
|
5
|
-
type: file
|
|
6
|
-
path_prefix: C:\Users\Abhishek Gupta\Desktop\github\embulk-filter-oracle_lookup\calendar.csv
|
|
7
|
-
parser:
|
|
8
|
-
type: csv
|
|
9
|
-
columns:
|
|
10
|
-
- { name: dim_calendar_key, type: long }
|
|
11
|
-
- { name: year_number, type: long }
|
|
12
|
-
- { name: quarter_number, type: long }
|
|
13
|
-
- { name: attr_1, type: string }
|
|
14
|
-
filters:
|
|
15
|
-
- type: oracle_lookup
|
|
16
|
-
driver_path: C:/Users/Abhishek Gupta/Desktop/mysql-connector-java-8.0.29-tidb-1.0.1.jar
|
|
17
|
-
driver_class: oracle.jdbc.driver.OracleDriver
|
|
18
|
-
host: localhost
|
|
19
|
-
port: 1521
|
|
20
|
-
database: XE
|
|
21
|
-
table: country
|
|
22
|
-
username: sys as sysdba
|
|
23
|
-
password: root
|
|
24
|
-
mapping_from:
|
|
25
|
-
- quarter_number
|
|
26
|
-
- attr_1
|
|
27
|
-
mapping_to:
|
|
28
|
-
- id
|
|
29
|
-
- country_address
|
|
30
|
-
new_columns:
|
|
31
|
-
- { name: country_name, type: string }
|
|
32
|
-
- { name: country_address, type: string }
|
|
33
|
-
out:
|
|
34
|
-
type: file
|
|
35
|
-
path_prefix: C:\Users\Abhishek Gupta\Desktop\A\output.csv
|
|
36
|
-
file_ext: csv
|
|
37
|
-
formatter:
|
|
38
|
-
type: csv
|
|
39
|
-
delimiter: "\t"
|
|
40
|
-
newline: CRLF
|
|
41
|
-
newline_in_field: LF
|
|
42
|
-
charset: UTF-8
|
|
43
|
-
quote_policy: MINIMAL
|
|
44
|
-
quote: '"'
|
|
45
|
-
escape: "\\"
|
|
46
|
-
null_string: "\\N"
|
|
47
|
-
default_timezone: 'UTC'
|