embulk-filter-postgres_lookup 0.1.6 → 0.1.7
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +82 -68
- data/build.gradle +1 -1
- data/classpath/embulk-filter-postgres_lookup-0.1.7.jar +0 -0
- data/example/postgres-filter-lookup_testing.yml.liquid +42 -0
- data/example/postgres-filter-lookup_testing_csv.yml.liquid +53 -0
- data/example/postgres-filter-lookup_testing_mssql.yml.liquid +41 -0
- data/example/postgres-filter-lookup_testing_mysql.yml.liquid +38 -0
- data/example/postgres-filter-lookup_testing_oracle.yml.liquid +43 -0
- data/src/main/java/org/embulk/filter/postgress_lookup/PostGresConnection.java +43 -40
- data/src/main/java/org/embulk/filter/postgress_lookup/PostgressLookupFilterPlugin.java +23 -1
- metadata +8 -5
- data/calendar.csv +0 -10
- data/classpath/embulk-filter-postgres_lookup-0.1.6.jar +0 -0
- data/example/config.yml +0 -48
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 8b9b5f5cd0836586e6ebf55891c634969983affdfbe4b0cfebd431a1f5262bd1
|
4
|
+
data.tar.gz: 7e7984515419c329e7fc86cd99b59578bf6e30f9437880a6a7a179dc2252d861
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 5bf4233c1fc1feef271222e48572560ea143e1a48feabab0838d7375aafd928675bac3bc054898207cf31f6f0e206f34f86beb0ef63184bf946b88f1043687bf
|
7
|
+
data.tar.gz: 34a1098823c389d4821e1faabcc56b8d29f5a5b600a093b3f67946490b14871c93a33135497bd68894f23d9191aa79ac4959d5427b0e4e2b817024452e298876
|
data/README.md
CHANGED
@@ -12,13 +12,13 @@ From strategy, to implementation, to ongoing managed services, Infoobjects creat
|
|
12
12
|
An Embulk filter plugin for Lookup Transformation with Postgres database
|
13
13
|
|
14
14
|
- **postgres_lookup**: Required attributes for the LookUp Filter Plugin -
|
15
|
-
- **driver_path**: driver path for your driver which you installed in your system (example
|
15
|
+
- **driver_path**: driver path for your driver which you installed in your system (example `....Path..../jarname.jar`)
|
16
16
|
- **driver_class**: driver class name (example `org.postgresql.Driver`)
|
17
17
|
- **schema_name**: schema name (example `testSchema`)
|
18
18
|
- **host**: database host (example `localhost`) (required)
|
19
19
|
- **port**: database port (example port for postgres `1433`) (required)
|
20
20
|
- **database**: database name (required)
|
21
|
-
- **
|
21
|
+
- **table_name**: table name of your database (required)
|
22
22
|
- **username**: username for your database (required)
|
23
23
|
- **password**: password for database (required)
|
24
24
|
- **mapping_from**: (Name of columns to be matched with table 2 columns) (required)
|
@@ -28,94 +28,108 @@ An Embulk filter plugin for Lookup Transformation with Postgres database
|
|
28
28
|
- **Name of column-1**: column name-1 from input file
|
29
29
|
- **Name of column-2**: column name-2 from input file
|
30
30
|
- **new_columns**: (New generated column names) (required)
|
31
|
-
|
32
|
-
|
31
|
+
- **Name-1,Type-1**: Any Name, Type of the name (name: pin, type: string)
|
32
|
+
- **Name-2,Type-2**: Any Name, Type of the name (name: gender, type: string)
|
33
|
+
- **Name-3,Type-3**: Any Name, Type of the name (name: phone_number, type: string) etc ...
|
33
34
|
## Example - columns
|
34
35
|
|
35
|
-
|
36
|
-
|
36
|
+
Customer.csv for table 1 is as follows :-
|
37
|
+
|
37
38
|
```
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
|
46
|
-
|
47
|
-
|
39
|
+
id customer_name address email car_name company
|
40
|
+
1 John Doe 123 Main St, Anytown USA john.doe@example.com Civic Honda
|
41
|
+
2 Jane Smith 456 Elm St, Anytown USA jane.smith@example.com E-Class Mercedes-Benz
|
42
|
+
3 Bob Johnson 789 Oak St, Anytown USA bob.johnson@example.com GLE-Class Mercedes-Benz
|
43
|
+
4 Amanda Hernandez 999 Cedar St, Anytown USA amanda.hernandez@example.com 911 119
|
44
|
+
5 Tom Brown 567 Pine St, Anytown USA tom.brown@example.com C-Class Mercedes-Benz
|
45
|
+
6 Samantha Davis 890 Cedar St, Anytown USA samantha.davis@example.com Civic Honda
|
46
|
+
7 Mike Wilson 1234 Spruce St, Anytown USA mike.wilson@example.com GLE-Class Mercedes-Benz
|
47
|
+
8 Jason Brown 888 Pine St, Anytown USA jason.brown@example.com 911 Porsche
|
48
|
+
9 David Rodriguez 9010 Oak St, Anytown USA david.rodriguez@example.com GLC-Class Mercedes-Benz
|
49
|
+
10 Mark Davis 666 Spruce St, Anytown USA mark.davis@example.com C-Class Mercedes-Benz
|
50
|
+
11 Chris Thompson 222 Cedar St, Anytown USA chris.thompson@example.com Cayenne Porsche
|
51
|
+
12 Linda Young 555 Birch St, Anytown USA linda.young@example.com RAV4
|
52
|
+
13 Kevin Hernandez 444 Maple St, Anytown USA kevin.hernandez@example.com 911 119
|
48
53
|
```
|
49
|
-
|
50
|
-
|
51
|
-
|
54
|
+
|
55
|
+
Car.csv for table 2 is as follows :-
|
56
|
+
|
52
57
|
```
|
53
|
-
|
54
|
-
|
55
|
-
|
56
|
-
|
57
|
-
|
58
|
-
|
59
|
-
|
60
|
-
|
61
|
-
|
62
|
-
|
63
|
-
|
64
|
-
|
65
|
-
|
58
|
+
car_id model brand category fuel_capacity
|
59
|
+
87 GLE-Class Mercedes-Benz SUV 80
|
60
|
+
101 Cayenne Porsche SUV 75
|
61
|
+
119 911 Porsche Sports Car 64
|
62
|
+
205 Accord Honda Sedan 56
|
63
|
+
334 Pilot Honda SUV 70
|
64
|
+
434 CR-v Honda SUV 64
|
65
|
+
559 C-Class Mercedes-Benz Sedan 66
|
66
|
+
603 Civic Honda Sedan 42
|
67
|
+
697 E-Class Mercedes-Benz Sedan 72
|
68
|
+
812 GLC-Class Mercedes-Benz Sedan 68
|
69
|
+
|
70
|
+
|
66
71
|
```
|
67
|
-
|
72
|
+
|
68
73
|
As shown in yaml below, columns mentioned in mapping_from will be mapped with columns mentioned in mapping_to
|
69
74
|
ie:
|
70
75
|
|
71
|
-
|
72
|
-
|
73
|
-
country_name : country_address
|
76
|
+
car_name : model
|
77
|
+
company : brand
|
74
78
|
|
75
|
-
After successful mapping an Output.csv file containing the columns mentioned in new_columns will be generated
|
76
|
-
|
77
|
-
|
79
|
+
After successful mapping an Output.csv file containing the columns mentioned in new_columns will be generated
|
78
80
|
|
79
81
|
Output File generated :-
|
80
|
-
|
82
|
+
|
81
83
|
```
|
82
|
-
|
83
|
-
|
84
|
-
|
85
|
-
|
86
|
-
|
87
|
-
|
88
|
-
|
89
|
-
|
90
|
-
|
91
|
-
|
84
|
+
id customer_name address email car_name company car_id category fuel_capacity
|
85
|
+
1 John Doe 123 Main St, Anytown USA john.doe@example.com Civic Honda 603 Sedan 42
|
86
|
+
2 Jane Smith 456 Elm St, Anytown USA jane.smith@example.com E-Class Mercedes-Benz 697 Sedan 72
|
87
|
+
3 Bob Johnson 789 Oak St, Anytown USA bob.johnson@example.com GLE-Class Mercedes-Benz 87 SUV 80
|
88
|
+
4 Amanda Hernandez 999 Cedar St, Anytown USA amanda.hernandez@example.com 911 119 0
|
89
|
+
5 Tom Brown 567 Pine St, Anytown USA tom.brown@example.com C-Class Mercedes-Benz 559 Sedan 66
|
90
|
+
6 Samantha Davis 890 Cedar St, Anytown USA samantha.davis@example.com Civic Honda 603 Sedan 42
|
91
|
+
7 Mike Wilson 1234 Spruce St, Anytown USA mike.wilson@example.com GLE-Class Mercedes-Benz 87 SUV 80
|
92
|
+
8 Jason Brown 888 Pine St, Anytown USA jason.brown@example.com 911 Porsche 119 Sport Car 64
|
93
|
+
9 David Rodriguez 9010 Oak St, Anytown USA david.rodriguez@example.com GLC-Class Mercedes-Benz 812 SUV 68
|
94
|
+
10 Mark Davis 666 Spruce St, Anytown USA mark.davis@example.com C-Class Mercedes-Benz 559 Sedan 66
|
95
|
+
11 Chris Thompson 222 Cedar St, Anytown USA chris.thompson@example.com Cayenne Porsche 101 SUV 75
|
96
|
+
12 Linda Young 555 Birch St, Anytown USA linda.young@example.com RAV4 \N 0
|
97
|
+
13 Kevin Hernandez 444 Maple St, Anytown USA kevin.hernandez@example.com 911 119 0
|
92
98
|
```
|
93
99
|
|
94
100
|
|
95
101
|
|
96
102
|
```yaml
|
97
|
-
|
98
|
-
|
99
|
-
|
100
|
-
|
101
|
-
|
102
|
-
|
103
|
-
|
104
|
-
|
105
|
-
|
106
|
-
|
107
|
-
|
108
|
-
|
109
|
-
|
110
|
-
|
111
|
-
|
112
|
-
|
113
|
-
|
103
|
+
filters:
|
104
|
+
- type: postgress_lookup
|
105
|
+
host: localhost
|
106
|
+
port: 5432
|
107
|
+
database: test
|
108
|
+
username: postgres
|
109
|
+
password: 1234
|
110
|
+
schema: public
|
111
|
+
table_name: car
|
112
|
+
mapping_from:
|
113
|
+
- car_name
|
114
|
+
- company
|
115
|
+
mapping_to:
|
116
|
+
- model
|
117
|
+
- brand
|
118
|
+
new_columns:
|
119
|
+
- { name: car_id, type: string }
|
120
|
+
- { name: category, type: string }
|
121
|
+
- { name: fuel_capacity, type: string }
|
114
122
|
```
|
115
123
|
|
116
124
|
Notes:
|
117
|
-
1. mapping_from attribute should be in same order as mentioned in input file.
|
125
|
+
1. mapping_from attribute should be in the same order as mentioned in the input file.
|
118
126
|
|
127
|
+
2. In case with JDBC plugin if any integer column returned as float/decimal then use to cast that column as long as below
|
128
|
+
```
|
129
|
+
column_options:
|
130
|
+
id: {value_type: long}
|
131
|
+
```
|
132
|
+
3. Matching columns data types must be int,long and String
|
119
133
|
## Development
|
120
134
|
|
121
135
|
Run example:
|
data/build.gradle
CHANGED
Binary file
|
@@ -0,0 +1,42 @@
|
|
1
|
+
exec:
|
2
|
+
max_threads: 1
|
3
|
+
min_output_tasks: 1
|
4
|
+
in:
|
5
|
+
type: postgresql
|
6
|
+
host: localhost
|
7
|
+
port: 5432
|
8
|
+
user: postgres
|
9
|
+
password: 1234
|
10
|
+
schema: public
|
11
|
+
database: test
|
12
|
+
table: customer
|
13
|
+
filters:
|
14
|
+
- type: postgress_lookup
|
15
|
+
host: localhost
|
16
|
+
port: 5432
|
17
|
+
database: test
|
18
|
+
username: postgres
|
19
|
+
password: 1234
|
20
|
+
schema: public
|
21
|
+
table_name: car
|
22
|
+
mapping_from:
|
23
|
+
- car_name
|
24
|
+
- company
|
25
|
+
mapping_to:
|
26
|
+
- model
|
27
|
+
- brand
|
28
|
+
new_columns:
|
29
|
+
- { name: car_id, type: string }
|
30
|
+
- { name: category, type: string }
|
31
|
+
- { name: fuel_capacity, type: string }
|
32
|
+
out:
|
33
|
+
type: postgresql
|
34
|
+
host: localhost
|
35
|
+
port: 5432
|
36
|
+
database: test
|
37
|
+
user: postgres
|
38
|
+
password: 1234
|
39
|
+
schema: public
|
40
|
+
table: output
|
41
|
+
mode: truncate_insert
|
42
|
+
|
@@ -0,0 +1,53 @@
|
|
1
|
+
exec:
|
2
|
+
max_threads: 1
|
3
|
+
min_output_tasks: 1
|
4
|
+
in:
|
5
|
+
type: file
|
6
|
+
path_prefix: "..path../customer.csv"
|
7
|
+
parser:
|
8
|
+
charset: UTF-8
|
9
|
+
type: csv
|
10
|
+
delimiter: ','
|
11
|
+
quote: '"'
|
12
|
+
header_line: true
|
13
|
+
columns:
|
14
|
+
- { name: id, type: long }
|
15
|
+
- { name: customer_name, type: string }
|
16
|
+
- { name: address, type: string }
|
17
|
+
- { name: email, type: string }
|
18
|
+
- { name: car_name, type: string }
|
19
|
+
- { name: company, type: string }
|
20
|
+
filters:
|
21
|
+
- type: postgress_lookup
|
22
|
+
host: localhost
|
23
|
+
port: 5432
|
24
|
+
database: test
|
25
|
+
username: postgres
|
26
|
+
password: 1234
|
27
|
+
schema: public
|
28
|
+
table_name: car
|
29
|
+
mapping_from:
|
30
|
+
- car_name
|
31
|
+
- company
|
32
|
+
mapping_to:
|
33
|
+
- model
|
34
|
+
- brand
|
35
|
+
new_columns:
|
36
|
+
- { name: car_id, type: string }
|
37
|
+
- { name: category, type: string }
|
38
|
+
- { name: fuel_capacity, type: string }
|
39
|
+
out:
|
40
|
+
type: file
|
41
|
+
path_prefix: "..path../output.csv"
|
42
|
+
file_ext: csv
|
43
|
+
formatter:
|
44
|
+
type: csv
|
45
|
+
delimiter: ","
|
46
|
+
newline: CRLF
|
47
|
+
newline_in_field: LF
|
48
|
+
charset: UTF-8
|
49
|
+
quote_policy: MINIMAL
|
50
|
+
quote: '"'
|
51
|
+
escape: "\\"
|
52
|
+
null_string: "\\N"
|
53
|
+
default_timezone: 'UTC'
|
@@ -0,0 +1,41 @@
|
|
1
|
+
exec:
|
2
|
+
max_threads: 1
|
3
|
+
min_output_tasks: 1
|
4
|
+
|
5
|
+
in:
|
6
|
+
type: sqlserver
|
7
|
+
host: Localhost
|
8
|
+
driver_path: "..path../mssql-jdbc-10.2.0.jre17.jar"
|
9
|
+
user: "usert"
|
10
|
+
password: "password"
|
11
|
+
database: "test"
|
12
|
+
table: customer
|
13
|
+
filters:
|
14
|
+
- type: postgress_lookup
|
15
|
+
host: localhost
|
16
|
+
port: 5432
|
17
|
+
database: test
|
18
|
+
username: postgres
|
19
|
+
password: 1234
|
20
|
+
schema: public
|
21
|
+
table_name: car
|
22
|
+
mapping_from:
|
23
|
+
- car_name
|
24
|
+
- company
|
25
|
+
mapping_to:
|
26
|
+
- model
|
27
|
+
- brand
|
28
|
+
new_columns:
|
29
|
+
- { name: car_id, type: string }
|
30
|
+
- { name: category, type: string }
|
31
|
+
- { name: fuel_capacity, type: string }
|
32
|
+
out:
|
33
|
+
type: sqlserver
|
34
|
+
host: Localhost
|
35
|
+
driver_path: "..path../mssql-jdbc-10.2.0.jre17.jar"
|
36
|
+
user: "usert"
|
37
|
+
password: "password"
|
38
|
+
database: "test"
|
39
|
+
table: output_table
|
40
|
+
mode: truncate_insert
|
41
|
+
|
@@ -0,0 +1,38 @@
|
|
1
|
+
exec:
|
2
|
+
max_threads: 1
|
3
|
+
min_output_tasks: 1
|
4
|
+
in:
|
5
|
+
type: mysql
|
6
|
+
host: localhost
|
7
|
+
user: root
|
8
|
+
password: 'passsword'
|
9
|
+
database: test
|
10
|
+
table: customer
|
11
|
+
filters:
|
12
|
+
- type: postgress_lookup
|
13
|
+
host: localhost
|
14
|
+
port: 5432
|
15
|
+
database: test
|
16
|
+
username: postgres
|
17
|
+
password: 1234
|
18
|
+
schema: public
|
19
|
+
table_name: car
|
20
|
+
mapping_from:
|
21
|
+
- car_name
|
22
|
+
- company
|
23
|
+
mapping_to:
|
24
|
+
- model
|
25
|
+
- brand
|
26
|
+
new_columns:
|
27
|
+
- { name: car_id, type: string }
|
28
|
+
- { name: category, type: string }
|
29
|
+
- { name: fuel_capacity, type: string }
|
30
|
+
out:
|
31
|
+
type: mysql
|
32
|
+
host: localhost
|
33
|
+
user: root
|
34
|
+
password: 'passsword'
|
35
|
+
database: test
|
36
|
+
table: output_table
|
37
|
+
mode: truncate_insert
|
38
|
+
|
@@ -0,0 +1,43 @@
|
|
1
|
+
exec:
|
2
|
+
max_threads: 1
|
3
|
+
min_output_tasks: 1
|
4
|
+
in:
|
5
|
+
type: jdbc
|
6
|
+
host: localhost
|
7
|
+
driver_path: "..path../ojdbc8.jar"
|
8
|
+
driver_class: 'oracle.jdbc.driver.OracleDriver'
|
9
|
+
url: jdbc:oracle:thin:@localhost:1521:orcl
|
10
|
+
user: MYUSER
|
11
|
+
password: ABCD
|
12
|
+
database: DEMO
|
13
|
+
table: customer
|
14
|
+
filters:
|
15
|
+
- type: postgress_lookup
|
16
|
+
host: localhost
|
17
|
+
port: 5432
|
18
|
+
database: test
|
19
|
+
username: postgres
|
20
|
+
password: 1234
|
21
|
+
schema: public
|
22
|
+
table_name: car
|
23
|
+
mapping_from:
|
24
|
+
- car_name
|
25
|
+
- company
|
26
|
+
mapping_to:
|
27
|
+
- model
|
28
|
+
- brand
|
29
|
+
new_columns:
|
30
|
+
- { name: car_id, type: string }
|
31
|
+
- { name: category, type: string }
|
32
|
+
- { name: fuel_capacity, type: string }
|
33
|
+
out:
|
34
|
+
type: jdbc
|
35
|
+
host: localhost
|
36
|
+
driver_path: "..path../ojdbc8.jar"
|
37
|
+
driver_class: 'oracle.jdbc.driver.OracleDriver'
|
38
|
+
url: jdbc:oracle:thin:@localhost:1521:orcl
|
39
|
+
user: MYUSER
|
40
|
+
password: ABCD
|
41
|
+
database: DEMO
|
42
|
+
table: output_table
|
43
|
+
mode: insert_direct
|
@@ -28,9 +28,9 @@ public class PostGresConnection {
|
|
28
28
|
try{
|
29
29
|
// Class.forName("org.postgresql.Driver");
|
30
30
|
if(task.getDriverClass().isPresent()){
|
31
|
-
this.
|
31
|
+
this.loadPostgressJdbcDriver(task.getDriverClass().get(),task.getDriverPath());
|
32
32
|
}else{
|
33
|
-
this.
|
33
|
+
this.loadPostgressJdbcDriver("org.postgresql.Driver",task.getDriverPath());
|
34
34
|
}
|
35
35
|
String url = "jdbc:postgresql://" + task.getHost() + ":"+task.getPort()+"/"+task.getDatabase();
|
36
36
|
connection= DriverManager.getConnection(url, task.getUserName(), task.getPassword());
|
@@ -58,68 +58,71 @@ public class PostGresConnection {
|
|
58
58
|
|
59
59
|
return connection;
|
60
60
|
}
|
61
|
-
private Class<? extends java.sql.Driver>
|
61
|
+
private Class<? extends java.sql.Driver> loadPostgressJdbcDriver(
|
62
62
|
final String className,
|
63
63
|
final Optional<String> driverPath)
|
64
64
|
{
|
65
|
-
synchronized (
|
66
|
-
if (
|
67
|
-
return
|
65
|
+
synchronized (postgressJdbcDriver) {
|
66
|
+
if (postgressJdbcDriver.get() != null) {
|
67
|
+
return postgressJdbcDriver.get();
|
68
|
+
}
|
69
|
+
if (driverPath.isPresent()) {
|
70
|
+
logger.info(
|
71
|
+
"\"driver_path\" is set to load the Postgress JDBC driver class \"{}\". Adding it to classpath.", className);
|
72
|
+
this.addDriverJarToClasspath(driverPath.get());
|
68
73
|
}
|
69
|
-
|
70
74
|
try {
|
71
75
|
// If the class is found from the ClassLoader of the plugin, that is prioritized the highest.
|
72
76
|
final Class<? extends java.sql.Driver> found = loadJdbcDriverClassForName(className);
|
73
|
-
|
77
|
+
postgressJdbcDriver.compareAndSet(null, found);
|
74
78
|
|
75
79
|
if (driverPath.isPresent()) {
|
76
80
|
logger.warn(
|
77
|
-
"\"driver_path\" is set while the
|
81
|
+
"\"driver_path\" is set while the Postgress JDBC driver class \"{}\" is found from the PluginClassLoader."
|
78
82
|
+ " \"driver_path\" is ignored.", className);
|
79
83
|
}
|
80
84
|
return found;
|
81
85
|
}
|
82
86
|
catch (final ClassNotFoundException ex) {
|
83
|
-
//
|
84
|
-
}
|
85
|
-
|
86
|
-
if (driverPath.isPresent()) {
|
87
|
-
logger.info(
|
88
|
-
"\"driver_path\" is set to load the MySQL JDBC driver class \"{}\". Adding it to classpath.", className);
|
89
|
-
this.addDriverJarToClasspath(driverPath.get());
|
87
|
+
//throw new ConfigException("The MySQL JDBC driver for the class \"" + className + "\" is not found.", ex);
|
90
88
|
}
|
91
|
-
|
92
|
-
|
93
|
-
|
94
|
-
|
95
|
-
|
96
|
-
|
97
|
-
|
98
|
-
return file.isFile() && file.getName().endsWith(".jar");
|
99
|
-
}
|
100
|
-
});
|
101
|
-
if (files == null || files.length == 0) {
|
102
|
-
throw new ConfigException(new ClassNotFoundException(
|
103
|
-
"The MySQL JDBC driver for the class \"" + className + "\" is not found"
|
104
|
-
+ " in \"default_jdbc_driver\" (" + root.getAbsolutePath() + ")."));
|
105
|
-
}
|
106
|
-
for (final File file : files) {
|
107
|
-
logger.info(
|
108
|
-
"The MySQL JDBC driver for the class \"{}\" is expected to be found"
|
109
|
-
+ " in \"default_jdbc_driver\" at {}.", className, file.getAbsolutePath());
|
110
|
-
this.addDriverJarToClasspath(file.getAbsolutePath());
|
89
|
+
final File root = this.findPluginRoot();
|
90
|
+
final File driverLib = new File(root, "default_jdbc_driver");
|
91
|
+
final File[] files = driverLib.listFiles(new FileFilter() {
|
92
|
+
@Override
|
93
|
+
public boolean accept(final File file)
|
94
|
+
{
|
95
|
+
return file.isFile() && file.getName().endsWith(".jar");
|
111
96
|
}
|
97
|
+
});
|
98
|
+
if (files == null || files.length == 0) {
|
99
|
+
throw new ConfigException(new ClassNotFoundException(
|
100
|
+
"The Postgress JDBC driver for the class \"" + className + "\" is not found"
|
101
|
+
+ " in \"default_jdbc_driver\" (" + root.getAbsolutePath() + ")."));
|
102
|
+
}
|
103
|
+
for (final File file : files) {
|
104
|
+
logger.info(
|
105
|
+
"The Postgress JDBC driver for the class \"{}\" is expected to be found"
|
106
|
+
+ " in \"default_jdbc_driver\" at {}.", className, file.getAbsolutePath());
|
107
|
+
this.addDriverJarToClasspath(file.getAbsolutePath());
|
112
108
|
}
|
113
109
|
|
114
110
|
try {
|
115
|
-
//
|
111
|
+
// If the class is found from the ClassLoader of the plugin, that is prioritized the highest.
|
116
112
|
final Class<? extends java.sql.Driver> found = loadJdbcDriverClassForName(className);
|
117
|
-
|
113
|
+
postgressJdbcDriver.compareAndSet(null, found);
|
114
|
+
|
115
|
+
if (driverPath.isPresent()) {
|
116
|
+
logger.warn(
|
117
|
+
"\"driver_path\" is set while the Postgress JDBC driver class \"{}\" is found from the PluginClassLoader."
|
118
|
+
+ " \"driver_path\" is ignored.", className);
|
119
|
+
}
|
118
120
|
return found;
|
119
121
|
}
|
120
122
|
catch (final ClassNotFoundException ex) {
|
121
|
-
throw new ConfigException("The
|
123
|
+
throw new ConfigException("The Postgress JDBC driver for the class \"" + className + "\" is not found.", ex);
|
122
124
|
}
|
125
|
+
|
123
126
|
}
|
124
127
|
}
|
125
128
|
|
@@ -129,7 +132,7 @@ public class PostGresConnection {
|
|
129
132
|
return (Class<? extends java.sql.Driver>) Class.forName(className);
|
130
133
|
}
|
131
134
|
|
132
|
-
private static final AtomicReference<Class<? extends Driver>>
|
135
|
+
private static final AtomicReference<Class<? extends Driver>> postgressJdbcDriver = new AtomicReference<>();
|
133
136
|
|
134
137
|
private static final Logger logger = LoggerFactory.getLogger(PostGresConnection.class);
|
135
138
|
|
@@ -6,6 +6,8 @@ import org.embulk.config.*;
|
|
6
6
|
import org.embulk.spi.*;
|
7
7
|
import org.embulk.spi.time.Timestamp;
|
8
8
|
import org.embulk.spi.type.Types;
|
9
|
+
import org.slf4j.Logger;
|
10
|
+
import org.slf4j.LoggerFactory;
|
9
11
|
|
10
12
|
import java.sql.*;
|
11
13
|
import java.time.Instant;
|
@@ -13,6 +15,8 @@ import java.util.*;
|
|
13
15
|
|
14
16
|
public class PostgressLookupFilterPlugin
|
15
17
|
implements FilterPlugin {
|
18
|
+
private static final Logger logger = LoggerFactory.getLogger(PostgressLookupFilterPlugin.class);
|
19
|
+
|
16
20
|
public interface PluginTask
|
17
21
|
extends Task {
|
18
22
|
|
@@ -25,7 +29,7 @@ public class PostgressLookupFilterPlugin
|
|
25
29
|
@Config("database")
|
26
30
|
public String getDatabase();
|
27
31
|
|
28
|
-
@Config("
|
32
|
+
@Config("table_name")
|
29
33
|
public String getTableName();
|
30
34
|
|
31
35
|
@Config("username")
|
@@ -208,6 +212,9 @@ public class PostgressLookupFilterPlugin
|
|
208
212
|
columnConfigList.add(columnConfig);
|
209
213
|
}
|
210
214
|
|
215
|
+
Set<String> unmatchedData = new LinkedHashSet<>();
|
216
|
+
List<String> keyColumns = task.getMappingFrom();
|
217
|
+
|
211
218
|
while (reader.nextRecord()) {
|
212
219
|
|
213
220
|
int colNum = 0;
|
@@ -240,6 +247,8 @@ public class PostgressLookupFilterPlugin
|
|
240
247
|
List<String> matchedData = new ArrayList<>();
|
241
248
|
if (keyValuePair.containsKey(key)) {
|
242
249
|
matchedData = keyValuePair.get(key);
|
250
|
+
}else{
|
251
|
+
unmatchedData.add(key);
|
243
252
|
}
|
244
253
|
|
245
254
|
if (matchedData.size() == 0) {
|
@@ -256,6 +265,19 @@ public class PostgressLookupFilterPlugin
|
|
256
265
|
builder.addRecord();
|
257
266
|
}
|
258
267
|
|
268
|
+
String info="\n--------------------Unmatched rows.....................\nMapping Key Columns: ";
|
269
|
+
for(int i=0;i<keyColumns.size();i++){
|
270
|
+
info+= keyColumns.get(i);
|
271
|
+
if(i!=keyColumns.size()-1){
|
272
|
+
info+=",";
|
273
|
+
}
|
274
|
+
}
|
275
|
+
info+="\n";
|
276
|
+
for(String key: unmatchedData){
|
277
|
+
info+= key+"\n";
|
278
|
+
}
|
279
|
+
logger.info(info);
|
280
|
+
|
259
281
|
}
|
260
282
|
|
261
283
|
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: embulk-filter-postgres_lookup
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.1.
|
4
|
+
version: 0.1.7
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Infoobjects Inc.
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2023-
|
11
|
+
date: 2023-03-03 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
@@ -49,14 +49,17 @@ files:
|
|
49
49
|
- LICENSE
|
50
50
|
- README.md
|
51
51
|
- build.gradle
|
52
|
-
-
|
53
|
-
- classpath/embulk-filter-postgres_lookup-0.1.6.jar
|
52
|
+
- classpath/embulk-filter-postgres_lookup-0.1.7.jar
|
54
53
|
- classpath/mysql-connector-java-8.0.19.jar
|
55
54
|
- classpath/postgresql-9.0-801.jdbc4.jar
|
56
55
|
- classpath/protobuf-java-3.6.1.jar
|
57
56
|
- config/checkstyle/checkstyle.xml
|
58
57
|
- config/checkstyle/default.xml
|
59
|
-
- example/
|
58
|
+
- example/postgres-filter-lookup_testing.yml.liquid
|
59
|
+
- example/postgres-filter-lookup_testing_csv.yml.liquid
|
60
|
+
- example/postgres-filter-lookup_testing_mssql.yml.liquid
|
61
|
+
- example/postgres-filter-lookup_testing_mysql.yml.liquid
|
62
|
+
- example/postgres-filter-lookup_testing_oracle.yml.liquid
|
60
63
|
- gradle/wrapper/gradle-wrapper.jar
|
61
64
|
- gradle/wrapper/gradle-wrapper.properties
|
62
65
|
- gradlew
|
data/calendar.csv
DELETED
@@ -1,10 +0,0 @@
|
|
1
|
-
dim_calendar_key,year_number,quarter_number,attr_1
|
2
|
-
-1,0,0
|
3
|
-
19900101,1990,1,FirstIndia
|
4
|
-
19900102,1990,2,SecondUSA
|
5
|
-
19900103,1990,,1000.56
|
6
|
-
19900104,1990,4,40_China
|
7
|
-
19900105,1990,5,50_Ukraine
|
8
|
-
19900106,1990,2,10_USA
|
9
|
-
19900107,1990,3,30_UK
|
10
|
-
19900109,1990,6,
|
Binary file
|
data/example/config.yml
DELETED
@@ -1,48 +0,0 @@
|
|
1
|
-
exec:
|
2
|
-
max_threads: 2
|
3
|
-
min_output_tasks: 1
|
4
|
-
in:
|
5
|
-
type: file
|
6
|
-
path_prefix: C:\Users\Abhishek Gupta\Desktop\github\embulk-filter-postgres_lookup\calendar.csv
|
7
|
-
parser:
|
8
|
-
type: csv
|
9
|
-
columns:
|
10
|
-
- { name: dim_calendar_key, type: long }
|
11
|
-
- { name: year_number, type: long }
|
12
|
-
- { name: quarter_number, type: long }
|
13
|
-
- { name: attr_1, type: string }
|
14
|
-
filters:
|
15
|
-
- type: postgress_lookup
|
16
|
-
driver_path: C:/Users/Abhishek Gupta/Desktop/postgresql-9.0-801.jdbc4.jar
|
17
|
-
driver_class: org.postgresql.Driver
|
18
|
-
schema_name: testSchema
|
19
|
-
host: localhost
|
20
|
-
port: 5432
|
21
|
-
database: Abhishek
|
22
|
-
table: country
|
23
|
-
username: postgres
|
24
|
-
password: root
|
25
|
-
mapping_from:
|
26
|
-
- quarter_number
|
27
|
-
- attr_1
|
28
|
-
mapping_to:
|
29
|
-
- id
|
30
|
-
- country_address
|
31
|
-
new_columns:
|
32
|
-
- { name: country_name, type: string }
|
33
|
-
- { name: country_address, type: string }
|
34
|
-
out:
|
35
|
-
type: file
|
36
|
-
path_prefix: C:\Users\Abhishek Gupta\Desktop\B\output.csv
|
37
|
-
file_ext: csv
|
38
|
-
formatter:
|
39
|
-
type: csv
|
40
|
-
delimiter: "\t"
|
41
|
-
newline: CRLF
|
42
|
-
newline_in_field: LF
|
43
|
-
charset: UTF-8
|
44
|
-
quote_policy: MINIMAL
|
45
|
-
quote: '"'
|
46
|
-
escape: "\\"
|
47
|
-
null_string: "\\N"
|
48
|
-
default_timezone: 'UTC'
|