embulk-filter-postgres_lookup 0.1.6 → 0.1.7

Sign up to get free protection for your applications and to get access to all the features.
Files changed (15) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +82 -68
  3. data/build.gradle +1 -1
  4. data/classpath/embulk-filter-postgres_lookup-0.1.7.jar +0 -0
  5. data/example/postgres-filter-lookup_testing.yml.liquid +42 -0
  6. data/example/postgres-filter-lookup_testing_csv.yml.liquid +53 -0
  7. data/example/postgres-filter-lookup_testing_mssql.yml.liquid +41 -0
  8. data/example/postgres-filter-lookup_testing_mysql.yml.liquid +38 -0
  9. data/example/postgres-filter-lookup_testing_oracle.yml.liquid +43 -0
  10. data/src/main/java/org/embulk/filter/postgress_lookup/PostGresConnection.java +43 -40
  11. data/src/main/java/org/embulk/filter/postgress_lookup/PostgressLookupFilterPlugin.java +23 -1
  12. metadata +8 -5
  13. data/calendar.csv +0 -10
  14. data/classpath/embulk-filter-postgres_lookup-0.1.6.jar +0 -0
  15. data/example/config.yml +0 -48
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 20b93f14dc975405be1a678e0646c142aec8b8e097478593d55b81de2f90010a
4
- data.tar.gz: 84ed7ed57d8d408630faf86380daf24d4e5b0a0636fc61c9cd3560b3486132af
3
+ metadata.gz: 8b9b5f5cd0836586e6ebf55891c634969983affdfbe4b0cfebd431a1f5262bd1
4
+ data.tar.gz: 7e7984515419c329e7fc86cd99b59578bf6e30f9437880a6a7a179dc2252d861
5
5
  SHA512:
6
- metadata.gz: b4d07337a0f56e160de416a79d836ad2804aedc3867a9393662b064dab628fd367c1d33d59b221654bb46ba68afdd1586e0880d9bee7f9406b357b63006bed3c
7
- data.tar.gz: d6d20a5856938c1a1b94a91408a6c9afe6def0408d18e799b5f1732ba63c2c9fbe1e1020c026710de3d7fa19b32460c3ce927b89e8849bbb3d0d2db3a54d2674
6
+ metadata.gz: 5bf4233c1fc1feef271222e48572560ea143e1a48feabab0838d7375aafd928675bac3bc054898207cf31f6f0e206f34f86beb0ef63184bf946b88f1043687bf
7
+ data.tar.gz: 34a1098823c389d4821e1faabcc56b8d29f5a5b600a093b3f67946490b14871c93a33135497bd68894f23d9191aa79ac4959d5427b0e4e2b817024452e298876
data/README.md CHANGED
@@ -12,13 +12,13 @@ From strategy, to implementation, to ongoing managed services, Infoobjects creat
12
12
  An Embulk filter plugin for Lookup Transformation with Postgres database
13
13
  ​
14
14
  - **postgres_lookup**: Required attributes for the LookUp Filter Plugin -
15
- - **driver_path**: driver path for your driver which you installed in your system (example `C:/Users/Abhishek Gupta/Desktop/postgresql-9.0-801.jdbc4.jar`)
15
+ - **driver_path**: driver path for your driver which you installed in your system (example `....Path..../jarname.jar`)
16
16
  - **driver_class**: driver class name (example `org.postgresql.Driver`)
17
17
  - **schema_name**: schema name (example `testSchema`)
18
18
  - **host**: database host (example `localhost`) (required)
19
19
  - **port**: database port (example port for postgres `1433`) (required)
20
20
  - **database**: database name (required)
21
- - **tablename**: table name of your database (required)
21
+ - **table_name**: table name of your database (required)
22
22
  - **username**: username for your database (required)
23
23
  - **password**: password for database (required)
24
24
  - **mapping_from**: (Name of columns to be matched with table 2 columns) (required)
@@ -28,94 +28,108 @@ An Embulk filter plugin for Lookup Transformation with Postgres database
28
28
  - **Name of column-1**: column name-1 from input file
29
29
  - **Name of column-2**: column name-2 from input file
30
30
  - **new_columns**: (New generated column names) (required)
31
- - **Name-1,Type-1**: Any Name, Type of the name (name: country_name, type: string)
32
- - **Name-2,Type-2**: Any Name, Type of the name (name: country_address, type: string) etc ...
31
+ - **Name-1,Type-1**: Any Name, Type of the name (name: pin, type: string)
32
+ - **Name-2,Type-2**: Any Name, Type of the name (name: gender, type: string)
33
+ - **Name-3,Type-3**: Any Name, Type of the name (name: phone_number, type: string) etc ...
33
34
  ## Example - columns
34
35
  ​
35
- Input1 for table 1 is as follows :-
36
- ​
36
+ Customer.csv for table 1 is as follows :-
37
+
37
38
  ```
38
- year country_code country_name literacy_rate
39
-
40
- 1990 1 India 80%
41
- 1993 2 USA 83%
42
- 1997 3 JAPAN
43
- 1999 4 China 72%
44
- 2000 5 Ukraine 68%
45
- 2002 6 Italy 79%
46
- 2004 7 UK 75%
47
- 2011 8 NULL 42%
39
+ id customer_name address email car_name company
40
+ 1 John Doe 123 Main St, Anytown USA john.doe@example.com Civic Honda
41
+ 2 Jane Smith 456 Elm St, Anytown USA jane.smith@example.com E-Class Mercedes-Benz
42
+ 3 Bob Johnson 789 Oak St, Anytown USA bob.johnson@example.com GLE-Class Mercedes-Benz
43
+ 4 Amanda Hernandez 999 Cedar St, Anytown USA amanda.hernandez@example.com 911 119
44
+ 5 Tom Brown 567 Pine St, Anytown USA tom.brown@example.com C-Class Mercedes-Benz
45
+ 6 Samantha Davis 890 Cedar St, Anytown USA samantha.davis@example.com Civic Honda
46
+ 7 Mike Wilson 1234 Spruce St, Anytown USA mike.wilson@example.com GLE-Class Mercedes-Benz
47
+ 8 Jason Brown 888 Pine St, Anytown USA jason.brown@example.com 911 Porsche
48
+ 9 David Rodriguez 9010 Oak St, Anytown USA david.rodriguez@example.com GLC-Class Mercedes-Benz
49
+ 10 Mark Davis 666 Spruce St, Anytown USA mark.davis@example.com C-Class Mercedes-Benz
50
+ 11 Chris Thompson 222 Cedar St, Anytown USA chris.thompson@example.com Cayenne Porsche
51
+ 12 Linda Young 555 Birch St, Anytown USA linda.young@example.com RAV4
52
+ 13 Kevin Hernandez 444 Maple St, Anytown USA kevin.hernandez@example.com 911 119
48
53
  ```
49
- ​
50
- Input2 for table 2 is as follows :-
51
- ​
54
+
55
+ Car.csv for table 2 is as follows :-
56
+
52
57
  ```
53
- id country_population country_address country_GDP
54
-
55
- 1 11.3 India 1.67
56
- 2 18.2 USA 16.72
57
- 3 30 JAPAN 5.00
58
- 4 4 China 9.33
59
- 5 57 Ukraine 1.08
60
- 6 63 Italy 2.068
61
- 7 17 UK 2.49
62
- 8 28 UAE 1.18
63
-
64
-
65
- Note: country_population is calculated in Billion and country_GDP is calculated in $USD Trillion
58
+ car_id model brand category fuel_capacity
59
+ 87 GLE-Class Mercedes-Benz SUV 80
60
+ 101 Cayenne Porsche SUV 75
61
+ 119 911 Porsche Sports Car 64
62
+ 205 Accord Honda Sedan 56
63
+ 334 Pilot Honda SUV 70
64
+ 434 CR-v Honda SUV 64
65
+ 559 C-Class Mercedes-Benz Sedan 66
66
+ 603 Civic Honda Sedan 42
67
+ 697 E-Class Mercedes-Benz Sedan 72
68
+ 812 GLC-Class Mercedes-Benz Sedan 68
69
+
70
+
66
71
  ```
67
- ​
72
+
68
73
  As shown in yaml below, columns mentioned in mapping_from will be mapped with columns mentioned in mapping_to
69
74
  ie:
70
75
 
71
- ​
72
- country_code : id
73
- country_name : country_address
76
+ car_name : model
77
+ company : brand
74
78
 
75
- After successful mapping an Output.csv file containing the columns mentioned in new_columns will be generated
76
- ​
77
- ​
79
+ After successful mapping an Output.csv file containing the columns mentioned in new_columns will be generated
78
80
 
79
81
  Output File generated :-
80
- ​
82
+
81
83
  ```
82
- year country_code country_name literacy_rate country_GDP country_population
83
-
84
- 1990 1 India 80% 1.67 11.3
85
- 1993 2 USA 83% 16.72 18.2
86
- 1997 3 JAPAN 5.00 30
87
- 1999 4 China 72% 9.33 4
88
- 2000 5 Ukraine 68% 1.08 57
89
- 2002 6 Italy 79% 2.068 63
90
- 2004 7 UK 75% 2.49 17
91
- 2011 8 NULL 42%
84
+ id customer_name address email car_name company car_id category fuel_capacity
85
+ 1 John Doe 123 Main St, Anytown USA john.doe@example.com Civic Honda 603 Sedan 42
86
+ 2 Jane Smith 456 Elm St, Anytown USA jane.smith@example.com E-Class Mercedes-Benz 697 Sedan 72
87
+ 3 Bob Johnson 789 Oak St, Anytown USA bob.johnson@example.com GLE-Class Mercedes-Benz 87 SUV 80
88
+ 4 Amanda Hernandez 999 Cedar St, Anytown USA amanda.hernandez@example.com 911 119 0
89
+ 5 Tom Brown 567 Pine St, Anytown USA tom.brown@example.com C-Class Mercedes-Benz 559 Sedan 66
90
+ 6 Samantha Davis 890 Cedar St, Anytown USA samantha.davis@example.com Civic Honda 603 Sedan 42
91
+ 7 Mike Wilson 1234 Spruce St, Anytown USA mike.wilson@example.com GLE-Class Mercedes-Benz 87 SUV 80
92
+ 8 Jason Brown 888 Pine St, Anytown USA jason.brown@example.com 911 Porsche 119 Sport Car 64
93
+ 9 David Rodriguez 9010 Oak St, Anytown USA david.rodriguez@example.com GLC-Class Mercedes-Benz 812 SUV 68
94
+ 10 Mark Davis 666 Spruce St, Anytown USA mark.davis@example.com C-Class Mercedes-Benz 559 Sedan 66
95
+ 11 Chris Thompson 222 Cedar St, Anytown USA chris.thompson@example.com Cayenne Porsche 101 SUV 75
96
+ 12 Linda Young 555 Birch St, Anytown USA linda.young@example.com RAV4 \N 0
97
+ 13 Kevin Hernandez 444 Maple St, Anytown USA kevin.hernandez@example.com 911 119 0
92
98
  ```
93
99
  ​
94
100
  ​
95
101
  ​
96
102
  ```yaml
97
- - type: postgress_lookup
98
- lookup_destination: mysql
99
- host: localhost
100
- port: 5432
101
- database: Abhishek
102
- tablename: country
103
- username: postgres
104
- password: root
105
- mapping_from:
106
- - quarter_number
107
- - attr_1
108
- mapping_to:
109
- - id
110
- - country_address
111
- new_columns:
112
- - { name: country_name, type: string }
113
- - { name: country_address, type: string }
103
+ filters:
104
+ - type: postgress_lookup
105
+ host: localhost
106
+ port: 5432
107
+ database: test
108
+ username: postgres
109
+ password: 1234
110
+ schema: public
111
+ table_name: car
112
+ mapping_from:
113
+ - car_name
114
+ - company
115
+ mapping_to:
116
+ - model
117
+ - brand
118
+ new_columns:
119
+ - { name: car_id, type: string }
120
+ - { name: category, type: string }
121
+ - { name: fuel_capacity, type: string }
114
122
  ```
115
123
  ​
116
124
  Notes:
117
- 1. mapping_from attribute should be in same order as mentioned in input file.
125
+ 1. mapping_from attribute should be in the same order as mentioned in the input file.
118
126
  ​
127
+ 2. In case with JDBC plugin if any integer column returned as float/decimal then use to cast that column as long as below
128
+ ```
129
+ column_options:
130
+ id: {value_type: long}
131
+ ```
132
+ 3. Matching columns data types must be int,long and String
119
133
  ## Development
120
134
  ​
121
135
  Run example:
data/build.gradle CHANGED
@@ -13,7 +13,7 @@ configurations {
13
13
  provided
14
14
  }
15
15
 
16
- version = "0.1.6"
16
+ version = "0.1.7"
17
17
 
18
18
  sourceCompatibility = 1.8
19
19
  targetCompatibility = 1.8
data/example/postgres-filter-lookup_testing.yml.liquid ADDED
@@ -0,0 +1,42 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+ in:
5
+ type: postgresql
6
+ host: localhost
7
+ port: 5432
8
+ user: postgres
9
+ password: 1234
10
+ schema: public
11
+ database: test
12
+ table: customer
13
+ filters:
14
+ - type: postgress_lookup
15
+ host: localhost
16
+ port: 5432
17
+ database: test
18
+ username: postgres
19
+ password: 1234
20
+ schema: public
21
+ table_name: car
22
+ mapping_from:
23
+ - car_name
24
+ - company
25
+ mapping_to:
26
+ - model
27
+ - brand
28
+ new_columns:
29
+ - { name: car_id, type: string }
30
+ - { name: category, type: string }
31
+ - { name: fuel_capacity, type: string }
32
+ out:
33
+ type: postgresql
34
+ host: localhost
35
+ port: 5432
36
+ database: test
37
+ user: postgres
38
+ password: 1234
39
+ schema: public
40
+ table: output
41
+ mode: truncate_insert
42
+
data/example/postgres-filter-lookup_testing_csv.yml.liquid ADDED
@@ -0,0 +1,53 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+ in:
5
+ type: file
6
+ path_prefix: "..path../customer.csv"
7
+ parser:
8
+ charset: UTF-8
9
+ type: csv
10
+ delimiter: ','
11
+ quote: '"'
12
+ header_line: true
13
+ columns:
14
+ - { name: id, type: long }
15
+ - { name: customer_name, type: string }
16
+ - { name: address, type: string }
17
+ - { name: email, type: string }
18
+ - { name: car_name, type: string }
19
+ - { name: company, type: string }
20
+ filters:
21
+ - type: postgress_lookup
22
+ host: localhost
23
+ port: 5432
24
+ database: test
25
+ username: postgres
26
+ password: 1234
27
+ schema: public
28
+ table_name: car
29
+ mapping_from:
30
+ - car_name
31
+ - company
32
+ mapping_to:
33
+ - model
34
+ - brand
35
+ new_columns:
36
+ - { name: car_id, type: string }
37
+ - { name: category, type: string }
38
+ - { name: fuel_capacity, type: string }
39
+ out:
40
+ type: file
41
+ path_prefix: "..path../output.csv"
42
+ file_ext: csv
43
+ formatter:
44
+ type: csv
45
+ delimiter: ","
46
+ newline: CRLF
47
+ newline_in_field: LF
48
+ charset: UTF-8
49
+ quote_policy: MINIMAL
50
+ quote: '"'
51
+ escape: "\\"
52
+ null_string: "\\N"
53
+ default_timezone: 'UTC'
data/example/postgres-filter-lookup_testing_mssql.yml.liquid ADDED
@@ -0,0 +1,41 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+
5
+ in:
6
+ type: sqlserver
7
+ host: Localhost
8
+ driver_path: "..path../mssql-jdbc-10.2.0.jre17.jar"
9
+ user: "usert"
10
+ password: "password"
11
+ database: "test"
12
+ table: customer
13
+ filters:
14
+ - type: postgress_lookup
15
+ host: localhost
16
+ port: 5432
17
+ database: test
18
+ username: postgres
19
+ password: 1234
20
+ schema: public
21
+ table_name: car
22
+ mapping_from:
23
+ - car_name
24
+ - company
25
+ mapping_to:
26
+ - model
27
+ - brand
28
+ new_columns:
29
+ - { name: car_id, type: string }
30
+ - { name: category, type: string }
31
+ - { name: fuel_capacity, type: string }
32
+ out:
33
+ type: sqlserver
34
+ host: Localhost
35
+ driver_path: "..path../mssql-jdbc-10.2.0.jre17.jar"
36
+ user: "usert"
37
+ password: "password"
38
+ database: "test"
39
+ table: output_table
40
+ mode: truncate_insert
41
+
data/example/postgres-filter-lookup_testing_mysql.yml.liquid ADDED
@@ -0,0 +1,38 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+ in:
5
+ type: mysql
6
+ host: localhost
7
+ user: root
8
+ password: 'passsword'
9
+ database: test
10
+ table: customer
11
+ filters:
12
+ - type: postgress_lookup
13
+ host: localhost
14
+ port: 5432
15
+ database: test
16
+ username: postgres
17
+ password: 1234
18
+ schema: public
19
+ table_name: car
20
+ mapping_from:
21
+ - car_name
22
+ - company
23
+ mapping_to:
24
+ - model
25
+ - brand
26
+ new_columns:
27
+ - { name: car_id, type: string }
28
+ - { name: category, type: string }
29
+ - { name: fuel_capacity, type: string }
30
+ out:
31
+ type: mysql
32
+ host: localhost
33
+ user: root
34
+ password: 'passsword'
35
+ database: test
36
+ table: output_table
37
+ mode: truncate_insert
38
+
data/example/postgres-filter-lookup_testing_oracle.yml.liquid ADDED
@@ -0,0 +1,43 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+ in:
5
+ type: jdbc
6
+ host: localhost
7
+ driver_path: "..path../ojdbc8.jar"
8
+ driver_class: 'oracle.jdbc.driver.OracleDriver'
9
+ url: jdbc:oracle:thin:@localhost:1521:orcl
10
+ user: MYUSER
11
+ password: ABCD
12
+ database: DEMO
13
+ table: customer
14
+ filters:
15
+ - type: postgress_lookup
16
+ host: localhost
17
+ port: 5432
18
+ database: test
19
+ username: postgres
20
+ password: 1234
21
+ schema: public
22
+ table_name: car
23
+ mapping_from:
24
+ - car_name
25
+ - company
26
+ mapping_to:
27
+ - model
28
+ - brand
29
+ new_columns:
30
+ - { name: car_id, type: string }
31
+ - { name: category, type: string }
32
+ - { name: fuel_capacity, type: string }
33
+ out:
34
+ type: jdbc
35
+ host: localhost
36
+ driver_path: "..path../ojdbc8.jar"
37
+ driver_class: 'oracle.jdbc.driver.OracleDriver'
38
+ url: jdbc:oracle:thin:@localhost:1521:orcl
39
+ user: MYUSER
40
+ password: ABCD
41
+ database: DEMO
42
+ table: output_table
43
+ mode: insert_direct
data/src/main/java/org/embulk/filter/postgress_lookup/PostGresConnection.java CHANGED
@@ -28,9 +28,9 @@ public class PostGresConnection {
28
28
  try{
29
29
  // Class.forName("org.postgresql.Driver");
30
30
  if(task.getDriverClass().isPresent()){
31
- this.loadMySqlJdbcDriver(task.getDriverClass().get(),task.getDriverPath());
31
+ this.loadPostgressJdbcDriver(task.getDriverClass().get(),task.getDriverPath());
32
32
  }else{
33
- this.loadMySqlJdbcDriver("org.postgresql.Driver",task.getDriverPath());
33
+ this.loadPostgressJdbcDriver("org.postgresql.Driver",task.getDriverPath());
34
34
  }
35
35
  String url = "jdbc:postgresql://" + task.getHost() + ":"+task.getPort()+"/"+task.getDatabase();
36
36
  connection= DriverManager.getConnection(url, task.getUserName(), task.getPassword());
@@ -58,68 +58,71 @@ public class PostGresConnection {
58
58
 
59
59
  return connection;
60
60
  }
61
- private Class<? extends java.sql.Driver> loadMySqlJdbcDriver(
61
+ private Class<? extends java.sql.Driver> loadPostgressJdbcDriver(
62
62
  final String className,
63
63
  final Optional<String> driverPath)
64
64
  {
65
- synchronized (mysqlJdbcDriver) {
66
- if (mysqlJdbcDriver.get() != null) {
67
- return mysqlJdbcDriver.get();
65
+ synchronized (postgressJdbcDriver) {
66
+ if (postgressJdbcDriver.get() != null) {
67
+ return postgressJdbcDriver.get();
68
+ }
69
+ if (driverPath.isPresent()) {
70
+ logger.info(
71
+ "\"driver_path\" is set to load the Postgress JDBC driver class \"{}\". Adding it to classpath.", className);
72
+ this.addDriverJarToClasspath(driverPath.get());
68
73
  }
69
-
70
74
  try {
71
75
  // If the class is found from the ClassLoader of the plugin, that is prioritized the highest.
72
76
  final Class<? extends java.sql.Driver> found = loadJdbcDriverClassForName(className);
73
- mysqlJdbcDriver.compareAndSet(null, found);
77
+ postgressJdbcDriver.compareAndSet(null, found);
74
78
 
75
79
  if (driverPath.isPresent()) {
76
80
  logger.warn(
77
- "\"driver_path\" is set while the MySQL JDBC driver class \"{}\" is found from the PluginClassLoader."
81
+ "\"driver_path\" is set while the Postgress JDBC driver class \"{}\" is found from the PluginClassLoader."
78
82
  + " \"driver_path\" is ignored.", className);
79
83
  }
80
84
  return found;
81
85
  }
82
86
  catch (final ClassNotFoundException ex) {
83
- // Pass-through once.
84
- }
85
-
86
- if (driverPath.isPresent()) {
87
- logger.info(
88
- "\"driver_path\" is set to load the MySQL JDBC driver class \"{}\". Adding it to classpath.", className);
89
- this.addDriverJarToClasspath(driverPath.get());
87
+ //throw new ConfigException("The MySQL JDBC driver for the class \"" + className + "\" is not found.", ex);
90
88
  }
91
- else {
92
- final File root = this.findPluginRoot();
93
- final File driverLib = new File(root, "default_jdbc_driver");
94
- final File[] files = driverLib.listFiles(new FileFilter() {
95
- @Override
96
- public boolean accept(final File file)
97
- {
98
- return file.isFile() && file.getName().endsWith(".jar");
99
- }
100
- });
101
- if (files == null || files.length == 0) {
102
- throw new ConfigException(new ClassNotFoundException(
103
- "The MySQL JDBC driver for the class \"" + className + "\" is not found"
104
- + " in \"default_jdbc_driver\" (" + root.getAbsolutePath() + ")."));
105
- }
106
- for (final File file : files) {
107
- logger.info(
108
- "The MySQL JDBC driver for the class \"{}\" is expected to be found"
109
- + " in \"default_jdbc_driver\" at {}.", className, file.getAbsolutePath());
110
- this.addDriverJarToClasspath(file.getAbsolutePath());
89
+ final File root = this.findPluginRoot();
90
+ final File driverLib = new File(root, "default_jdbc_driver");
91
+ final File[] files = driverLib.listFiles(new FileFilter() {
92
+ @Override
93
+ public boolean accept(final File file)
94
+ {
95
+ return file.isFile() && file.getName().endsWith(".jar");
111
96
  }
97
+ });
98
+ if (files == null || files.length == 0) {
99
+ throw new ConfigException(new ClassNotFoundException(
100
+ "The Postgress JDBC driver for the class \"" + className + "\" is not found"
101
+ + " in \"default_jdbc_driver\" (" + root.getAbsolutePath() + ")."));
102
+ }
103
+ for (final File file : files) {
104
+ logger.info(
105
+ "The Postgress JDBC driver for the class \"{}\" is expected to be found"
106
+ + " in \"default_jdbc_driver\" at {}.", className, file.getAbsolutePath());
107
+ this.addDriverJarToClasspath(file.getAbsolutePath());
112
108
  }
113
109
 
114
110
  try {
115
- // Retrying to find the class from the ClassLoader of the plugin.
111
+ // If the class is found from the ClassLoader of the plugin, that is prioritized the highest.
116
112
  final Class<? extends java.sql.Driver> found = loadJdbcDriverClassForName(className);
117
- mysqlJdbcDriver.compareAndSet(null, found);
113
+ postgressJdbcDriver.compareAndSet(null, found);
114
+
115
+ if (driverPath.isPresent()) {
116
+ logger.warn(
117
+ "\"driver_path\" is set while the Postgress JDBC driver class \"{}\" is found from the PluginClassLoader."
118
+ + " \"driver_path\" is ignored.", className);
119
+ }
118
120
  return found;
119
121
  }
120
122
  catch (final ClassNotFoundException ex) {
121
- throw new ConfigException("The MySQL JDBC driver for the class \"" + className + "\" is not found.", ex);
123
+ throw new ConfigException("The Postgress JDBC driver for the class \"" + className + "\" is not found.", ex);
122
124
  }
125
+
123
126
  }
124
127
  }
125
128
 
@@ -129,7 +132,7 @@ public class PostGresConnection {
129
132
  return (Class<? extends java.sql.Driver>) Class.forName(className);
130
133
  }
131
134
 
132
- private static final AtomicReference<Class<? extends Driver>> mysqlJdbcDriver = new AtomicReference<>();
135
+ private static final AtomicReference<Class<? extends Driver>> postgressJdbcDriver = new AtomicReference<>();
133
136
 
134
137
  private static final Logger logger = LoggerFactory.getLogger(PostGresConnection.class);
135
138
 
data/src/main/java/org/embulk/filter/postgress_lookup/PostgressLookupFilterPlugin.java CHANGED
@@ -6,6 +6,8 @@ import org.embulk.config.*;
6
6
  import org.embulk.spi.*;
7
7
  import org.embulk.spi.time.Timestamp;
8
8
  import org.embulk.spi.type.Types;
9
+ import org.slf4j.Logger;
10
+ import org.slf4j.LoggerFactory;
9
11
 
10
12
  import java.sql.*;
11
13
  import java.time.Instant;
@@ -13,6 +15,8 @@ import java.util.*;
13
15
 
14
16
  public class PostgressLookupFilterPlugin
15
17
  implements FilterPlugin {
18
+ private static final Logger logger = LoggerFactory.getLogger(PostgressLookupFilterPlugin.class);
19
+
16
20
  public interface PluginTask
17
21
  extends Task {
18
22
 
@@ -25,7 +29,7 @@ public class PostgressLookupFilterPlugin
25
29
  @Config("database")
26
30
  public String getDatabase();
27
31
 
28
- @Config("table")
32
+ @Config("table_name")
29
33
  public String getTableName();
30
34
 
31
35
  @Config("username")
@@ -208,6 +212,9 @@ public class PostgressLookupFilterPlugin
208
212
  columnConfigList.add(columnConfig);
209
213
  }
210
214
 
215
+ Set<String> unmatchedData = new LinkedHashSet<>();
216
+ List<String> keyColumns = task.getMappingFrom();
217
+
211
218
  while (reader.nextRecord()) {
212
219
 
213
220
  int colNum = 0;
@@ -240,6 +247,8 @@ public class PostgressLookupFilterPlugin
240
247
  List<String> matchedData = new ArrayList<>();
241
248
  if (keyValuePair.containsKey(key)) {
242
249
  matchedData = keyValuePair.get(key);
250
+ }else{
251
+ unmatchedData.add(key);
243
252
  }
244
253
 
245
254
  if (matchedData.size() == 0) {
@@ -256,6 +265,19 @@ public class PostgressLookupFilterPlugin
256
265
  builder.addRecord();
257
266
  }
258
267
 
268
+ String info="\n--------------------Unmatched rows.....................\nMapping Key Columns: ";
269
+ for(int i=0;i<keyColumns.size();i++){
270
+ info+= keyColumns.get(i);
271
+ if(i!=keyColumns.size()-1){
272
+ info+=",";
273
+ }
274
+ }
275
+ info+="\n";
276
+ for(String key: unmatchedData){
277
+ info+= key+"\n";
278
+ }
279
+ logger.info(info);
280
+
259
281
  }
260
282
 
261
283
 
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: embulk-filter-postgres_lookup
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.6
4
+ version: 0.1.7
5
5
  platform: ruby
6
6
  authors:
7
7
  - Infoobjects Inc.
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2023-02-10 00:00:00.000000000 Z
11
+ date: 2023-03-03 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
@@ -49,14 +49,17 @@ files:
49
49
  - LICENSE
50
50
  - README.md
51
51
  - build.gradle
52
- - calendar.csv
53
- - classpath/embulk-filter-postgres_lookup-0.1.6.jar
52
+ - classpath/embulk-filter-postgres_lookup-0.1.7.jar
54
53
  - classpath/mysql-connector-java-8.0.19.jar
55
54
  - classpath/postgresql-9.0-801.jdbc4.jar
56
55
  - classpath/protobuf-java-3.6.1.jar
57
56
  - config/checkstyle/checkstyle.xml
58
57
  - config/checkstyle/default.xml
59
- - example/config.yml
58
+ - example/postgres-filter-lookup_testing.yml.liquid
59
+ - example/postgres-filter-lookup_testing_csv.yml.liquid
60
+ - example/postgres-filter-lookup_testing_mssql.yml.liquid
61
+ - example/postgres-filter-lookup_testing_mysql.yml.liquid
62
+ - example/postgres-filter-lookup_testing_oracle.yml.liquid
60
63
  - gradle/wrapper/gradle-wrapper.jar
61
64
  - gradle/wrapper/gradle-wrapper.properties
62
65
  - gradlew
data/calendar.csv DELETED
@@ -1,10 +0,0 @@
1
- dim_calendar_key,year_number,quarter_number,attr_1
2
- -1,0,0
3
- 19900101,1990,1,FirstIndia
4
- 19900102,1990,2,SecondUSA
5
- 19900103,1990,,1000.56
6
- 19900104,1990,4,40_China
7
- 19900105,1990,5,50_Ukraine
8
- 19900106,1990,2,10_USA
9
- 19900107,1990,3,30_UK
10
- 19900109,1990,6,
data/example/config.yml DELETED
@@ -1,48 +0,0 @@
1
- exec:
2
- max_threads: 2
3
- min_output_tasks: 1
4
- in:
5
- type: file
6
- path_prefix: C:\Users\Abhishek Gupta\Desktop\github\embulk-filter-postgres_lookup\calendar.csv
7
- parser:
8
- type: csv
9
- columns:
10
- - { name: dim_calendar_key, type: long }
11
- - { name: year_number, type: long }
12
- - { name: quarter_number, type: long }
13
- - { name: attr_1, type: string }
14
- filters:
15
- - type: postgress_lookup
16
- driver_path: C:/Users/Abhishek Gupta/Desktop/postgresql-9.0-801.jdbc4.jar
17
- driver_class: org.postgresql.Driver
18
- schema_name: testSchema
19
- host: localhost
20
- port: 5432
21
- database: Abhishek
22
- table: country
23
- username: postgres
24
- password: root
25
- mapping_from:
26
- - quarter_number
27
- - attr_1
28
- mapping_to:
29
- - id
30
- - country_address
31
- new_columns:
32
- - { name: country_name, type: string }
33
- - { name: country_address, type: string }
34
- out:
35
- type: file
36
- path_prefix: C:\Users\Abhishek Gupta\Desktop\B\output.csv
37
- file_ext: csv
38
- formatter:
39
- type: csv
40
- delimiter: "\t"
41
- newline: CRLF
42
- newline_in_field: LF
43
- charset: UTF-8
44
- quote_policy: MINIMAL
45
- quote: '"'
46
- escape: "\\"
47
- null_string: "\\N"
48
- default_timezone: 'UTC'