embulk-filter-postgres_lookup 0.1.6 → 0.1.7

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (15) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +82 -68
  3. data/build.gradle +1 -1
  4. data/classpath/embulk-filter-postgres_lookup-0.1.7.jar +0 -0
  5. data/example/postgres-filter-lookup_testing.yml.liquid +42 -0
  6. data/example/postgres-filter-lookup_testing_csv.yml.liquid +53 -0
  7. data/example/postgres-filter-lookup_testing_mssql.yml.liquid +41 -0
  8. data/example/postgres-filter-lookup_testing_mysql.yml.liquid +38 -0
  9. data/example/postgres-filter-lookup_testing_oracle.yml.liquid +43 -0
  10. data/src/main/java/org/embulk/filter/postgress_lookup/PostGresConnection.java +43 -40
  11. data/src/main/java/org/embulk/filter/postgress_lookup/PostgressLookupFilterPlugin.java +23 -1
  12. metadata +8 -5
  13. data/calendar.csv +0 -10
  14. data/classpath/embulk-filter-postgres_lookup-0.1.6.jar +0 -0
  15. data/example/config.yml +0 -48
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 20b93f14dc975405be1a678e0646c142aec8b8e097478593d55b81de2f90010a
4
- data.tar.gz: 84ed7ed57d8d408630faf86380daf24d4e5b0a0636fc61c9cd3560b3486132af
3
+ metadata.gz: 8b9b5f5cd0836586e6ebf55891c634969983affdfbe4b0cfebd431a1f5262bd1
4
+ data.tar.gz: 7e7984515419c329e7fc86cd99b59578bf6e30f9437880a6a7a179dc2252d861
5
5
  SHA512:
6
- metadata.gz: b4d07337a0f56e160de416a79d836ad2804aedc3867a9393662b064dab628fd367c1d33d59b221654bb46ba68afdd1586e0880d9bee7f9406b357b63006bed3c
7
- data.tar.gz: d6d20a5856938c1a1b94a91408a6c9afe6def0408d18e799b5f1732ba63c2c9fbe1e1020c026710de3d7fa19b32460c3ce927b89e8849bbb3d0d2db3a54d2674
6
+ metadata.gz: 5bf4233c1fc1feef271222e48572560ea143e1a48feabab0838d7375aafd928675bac3bc054898207cf31f6f0e206f34f86beb0ef63184bf946b88f1043687bf
7
+ data.tar.gz: 34a1098823c389d4821e1faabcc56b8d29f5a5b600a093b3f67946490b14871c93a33135497bd68894f23d9191aa79ac4959d5427b0e4e2b817024452e298876
data/README.md CHANGED
@@ -12,13 +12,13 @@ From strategy, to implementation, to ongoing managed services, Infoobjects creat
12
12
  An Embulk filter plugin for Lookup Transformation with Postgres database
13
13
  ​
14
14
  - **postgres_lookup**: Required attributes for the LookUp Filter Plugin -
15
- - **driver_path**: driver path for your driver which you installed in your system (example `C:/Users/Abhishek Gupta/Desktop/postgresql-9.0-801.jdbc4.jar`)
15
+ - **driver_path**: driver path for your driver which you installed in your system (example `....Path..../jarname.jar`)
16
16
  - **driver_class**: driver class name (example `org.postgresql.Driver`)
17
17
  - **schema_name**: schema name (example `testSchema`)
18
18
  - **host**: database host (example `localhost`) (required)
19
19
  - **port**: database port (example port for postgres `1433`) (required)
20
20
  - **database**: database name (required)
21
- - **tablename**: table name of your database (required)
21
+ - **table_name**: table name of your database (required)
22
22
  - **username**: username for your database (required)
23
23
  - **password**: password for database (required)
24
24
  - **mapping_from**: (Name of columns to be matched with table 2 columns) (required)
@@ -28,94 +28,108 @@ An Embulk filter plugin for Lookup Transformation with Postgres database
28
28
  - **Name of column-1**: column name-1 from input file
29
29
  - **Name of column-2**: column name-2 from input file
30
30
  - **new_columns**: (New generated column names) (required)
31
- - **Name-1,Type-1**: Any Name, Type of the name (name: country_name, type: string)
32
- - **Name-2,Type-2**: Any Name, Type of the name (name: country_address, type: string) etc ...
31
+ - **Name-1,Type-1**: Any Name, Type of the name (name: pin, type: string)
32
+ - **Name-2,Type-2**: Any Name, Type of the name (name: gender, type: string)
33
+ - **Name-3,Type-3**: Any Name, Type of the name (name: phone_number, type: string) etc ...
33
34
  ## Example - columns
34
35
  ​
35
- Input1 for table 1 is as follows :-
36
- ​
36
+ Customer.csv for table 1 is as follows :-
37
+
37
38
  ```
38
- year country_code country_name literacy_rate
39
-
40
- 1990 1 India 80%
41
- 1993 2 USA 83%
42
- 1997 3 JAPAN
43
- 1999 4 China 72%
44
- 2000 5 Ukraine 68%
45
- 2002 6 Italy 79%
46
- 2004 7 UK 75%
47
- 2011 8 NULL 42%
39
+ id customer_name address email car_name company
40
+ 1 John Doe 123 Main St, Anytown USA john.doe@example.com Civic Honda
41
+ 2 Jane Smith 456 Elm St, Anytown USA jane.smith@example.com E-Class Mercedes-Benz
42
+ 3 Bob Johnson 789 Oak St, Anytown USA bob.johnson@example.com GLE-Class Mercedes-Benz
43
+ 4 Amanda Hernandez 999 Cedar St, Anytown USA amanda.hernandez@example.com 911 119
44
+ 5 Tom Brown 567 Pine St, Anytown USA tom.brown@example.com C-Class Mercedes-Benz
45
+ 6 Samantha Davis 890 Cedar St, Anytown USA samantha.davis@example.com Civic Honda
46
+ 7 Mike Wilson 1234 Spruce St, Anytown USA mike.wilson@example.com GLE-Class Mercedes-Benz
47
+ 8 Jason Brown 888 Pine St, Anytown USA jason.brown@example.com 911 Porsche
48
+ 9 David Rodriguez 9010 Oak St, Anytown USA david.rodriguez@example.com GLC-Class Mercedes-Benz
49
+ 10 Mark Davis 666 Spruce St, Anytown USA mark.davis@example.com C-Class Mercedes-Benz
50
+ 11 Chris Thompson 222 Cedar St, Anytown USA chris.thompson@example.com Cayenne Porsche
51
+ 12 Linda Young 555 Birch St, Anytown USA linda.young@example.com RAV4
52
+ 13 Kevin Hernandez 444 Maple St, Anytown USA kevin.hernandez@example.com 911 119
48
53
  ```
49
- ​
50
- Input2 for table 2 is as follows :-
51
- ​
54
+
55
+ Car.csv for table 2 is as follows :-
56
+
52
57
  ```
53
- id country_population country_address country_GDP
54
-
55
- 1 11.3 India 1.67
56
- 2 18.2 USA 16.72
57
- 3 30 JAPAN 5.00
58
- 4 4 China 9.33
59
- 5 57 Ukraine 1.08
60
- 6 63 Italy 2.068
61
- 7 17 UK 2.49
62
- 8 28 UAE 1.18
63
-
64
-
65
- Note: country_population is calculated in Billion and country_GDP is calculated in $USD Trillion
58
+ car_id model brand category fuel_capacity
59
+ 87 GLE-Class Mercedes-Benz SUV 80
60
+ 101 Cayenne Porsche SUV 75
61
+ 119 911 Porsche Sports Car 64
62
+ 205 Accord Honda Sedan 56
63
+ 334 Pilot Honda SUV 70
64
+ 434 CR-v Honda SUV 64
65
+ 559 C-Class Mercedes-Benz Sedan 66
66
+ 603 Civic Honda Sedan 42
67
+ 697 E-Class Mercedes-Benz Sedan 72
68
+ 812 GLC-Class Mercedes-Benz Sedan 68
69
+
70
+
66
71
  ```
67
- ​
72
+
68
73
  As shown in yaml below, columns mentioned in mapping_from will be mapped with columns mentioned in mapping_to
69
74
  ie:
70
75
 
71
- ​
72
- country_code : id
73
- country_name : country_address
76
+ car_name : model
77
+ company : brand
74
78
 
75
- After successful mapping an Output.csv file containing the columns mentioned in new_columns will be generated
76
- ​
77
- ​
79
+ After successful mapping an Output.csv file containing the columns mentioned in new_columns will be generated
78
80
 
79
81
  Output File generated :-
80
- ​
82
+
81
83
  ```
82
- year country_code country_name literacy_rate country_GDP country_population
83
-
84
- 1990 1 India 80% 1.67 11.3
85
- 1993 2 USA 83% 16.72 18.2
86
- 1997 3 JAPAN 5.00 30
87
- 1999 4 China 72% 9.33 4
88
- 2000 5 Ukraine 68% 1.08 57
89
- 2002 6 Italy 79% 2.068 63
90
- 2004 7 UK 75% 2.49 17
91
- 2011 8 NULL 42%
84
+ id customer_name address email car_name company car_id category fuel_capacity
85
+ 1 John Doe 123 Main St, Anytown USA john.doe@example.com Civic Honda 603 Sedan 42
86
+ 2 Jane Smith 456 Elm St, Anytown USA jane.smith@example.com E-Class Mercedes-Benz 697 Sedan 72
87
+ 3 Bob Johnson 789 Oak St, Anytown USA bob.johnson@example.com GLE-Class Mercedes-Benz 87 SUV 80
88
+ 4 Amanda Hernandez 999 Cedar St, Anytown USA amanda.hernandez@example.com 911 119 0
89
+ 5 Tom Brown 567 Pine St, Anytown USA tom.brown@example.com C-Class Mercedes-Benz 559 Sedan 66
90
+ 6 Samantha Davis 890 Cedar St, Anytown USA samantha.davis@example.com Civic Honda 603 Sedan 42
91
+ 7 Mike Wilson 1234 Spruce St, Anytown USA mike.wilson@example.com GLE-Class Mercedes-Benz 87 SUV 80
92
+ 8 Jason Brown 888 Pine St, Anytown USA jason.brown@example.com 911 Porsche 119 Sport Car 64
93
+ 9 David Rodriguez 9010 Oak St, Anytown USA david.rodriguez@example.com GLC-Class Mercedes-Benz 812 SUV 68
94
+ 10 Mark Davis 666 Spruce St, Anytown USA mark.davis@example.com C-Class Mercedes-Benz 559 Sedan 66
95
+ 11 Chris Thompson 222 Cedar St, Anytown USA chris.thompson@example.com Cayenne Porsche 101 SUV 75
96
+ 12 Linda Young 555 Birch St, Anytown USA linda.young@example.com RAV4 \N 0
97
+ 13 Kevin Hernandez 444 Maple St, Anytown USA kevin.hernandez@example.com 911 119 0
92
98
  ```
93
99
  ​
94
100
  ​
95
101
  ​
96
102
  ```yaml
97
- - type: postgress_lookup
98
- lookup_destination: mysql
99
- host: localhost
100
- port: 5432
101
- database: Abhishek
102
- tablename: country
103
- username: postgres
104
- password: root
105
- mapping_from:
106
- - quarter_number
107
- - attr_1
108
- mapping_to:
109
- - id
110
- - country_address
111
- new_columns:
112
- - { name: country_name, type: string }
113
- - { name: country_address, type: string }
103
+ filters:
104
+ - type: postgress_lookup
105
+ host: localhost
106
+ port: 5432
107
+ database: test
108
+ username: postgres
109
+ password: 1234
110
+ schema: public
111
+ table_name: car
112
+ mapping_from:
113
+ - car_name
114
+ - company
115
+ mapping_to:
116
+ - model
117
+ - brand
118
+ new_columns:
119
+ - { name: car_id, type: string }
120
+ - { name: category, type: string }
121
+ - { name: fuel_capacity, type: string }
114
122
  ```
115
123
  ​
116
124
  Notes:
117
- 1. mapping_from attribute should be in same order as mentioned in input file.
125
+ 1. mapping_from attribute should be in the same order as mentioned in the input file.
118
126
  ​
127
+ 2. In case with JDBC plugin if any integer column returned as float/decimal then use to cast that column as long as below
128
+ ```
129
+ column_options:
130
+ id: {value_type: long}
131
+ ```
132
+ 3. Matching columns data types must be int,long and String
119
133
  ## Development
120
134
  ​
121
135
  Run example:
data/build.gradle CHANGED
@@ -13,7 +13,7 @@ configurations {
13
13
  provided
14
14
  }
15
15
 
16
- version = "0.1.6"
16
+ version = "0.1.7"
17
17
 
18
18
  sourceCompatibility = 1.8
19
19
  targetCompatibility = 1.8
data/example/postgres-filter-lookup_testing.yml.liquid ADDED
@@ -0,0 +1,42 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+ in:
5
+ type: postgresql
6
+ host: localhost
7
+ port: 5432
8
+ user: postgres
9
+ password: 1234
10
+ schema: public
11
+ database: test
12
+ table: customer
13
+ filters:
14
+ - type: postgress_lookup
15
+ host: localhost
16
+ port: 5432
17
+ database: test
18
+ username: postgres
19
+ password: 1234
20
+ schema: public
21
+ table_name: car
22
+ mapping_from:
23
+ - car_name
24
+ - company
25
+ mapping_to:
26
+ - model
27
+ - brand
28
+ new_columns:
29
+ - { name: car_id, type: string }
30
+ - { name: category, type: string }
31
+ - { name: fuel_capacity, type: string }
32
+ out:
33
+ type: postgresql
34
+ host: localhost
35
+ port: 5432
36
+ database: test
37
+ user: postgres
38
+ password: 1234
39
+ schema: public
40
+ table: output
41
+ mode: truncate_insert
42
+
data/example/postgres-filter-lookup_testing_csv.yml.liquid ADDED
@@ -0,0 +1,53 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+ in:
5
+ type: file
6
+ path_prefix: "..path../customer.csv"
7
+ parser:
8
+ charset: UTF-8
9
+ type: csv
10
+ delimiter: ','
11
+ quote: '"'
12
+ header_line: true
13
+ columns:
14
+ - { name: id, type: long }
15
+ - { name: customer_name, type: string }
16
+ - { name: address, type: string }
17
+ - { name: email, type: string }
18
+ - { name: car_name, type: string }
19
+ - { name: company, type: string }
20
+ filters:
21
+ - type: postgress_lookup
22
+ host: localhost
23
+ port: 5432
24
+ database: test
25
+ username: postgres
26
+ password: 1234
27
+ schema: public
28
+ table_name: car
29
+ mapping_from:
30
+ - car_name
31
+ - company
32
+ mapping_to:
33
+ - model
34
+ - brand
35
+ new_columns:
36
+ - { name: car_id, type: string }
37
+ - { name: category, type: string }
38
+ - { name: fuel_capacity, type: string }
39
+ out:
40
+ type: file
41
+ path_prefix: "..path../output.csv"
42
+ file_ext: csv
43
+ formatter:
44
+ type: csv
45
+ delimiter: ","
46
+ newline: CRLF
47
+ newline_in_field: LF
48
+ charset: UTF-8
49
+ quote_policy: MINIMAL
50
+ quote: '"'
51
+ escape: "\\"
52
+ null_string: "\\N"
53
+ default_timezone: 'UTC'
data/example/postgres-filter-lookup_testing_mssql.yml.liquid ADDED
@@ -0,0 +1,41 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+
5
+ in:
6
+ type: sqlserver
7
+ host: Localhost
8
+ driver_path: "..path../mssql-jdbc-10.2.0.jre17.jar"
9
+ user: "usert"
10
+ password: "password"
11
+ database: "test"
12
+ table: customer
13
+ filters:
14
+ - type: postgress_lookup
15
+ host: localhost
16
+ port: 5432
17
+ database: test
18
+ username: postgres
19
+ password: 1234
20
+ schema: public
21
+ table_name: car
22
+ mapping_from:
23
+ - car_name
24
+ - company
25
+ mapping_to:
26
+ - model
27
+ - brand
28
+ new_columns:
29
+ - { name: car_id, type: string }
30
+ - { name: category, type: string }
31
+ - { name: fuel_capacity, type: string }
32
+ out:
33
+ type: sqlserver
34
+ host: Localhost
35
+ driver_path: "..path../mssql-jdbc-10.2.0.jre17.jar"
36
+ user: "usert"
37
+ password: "password"
38
+ database: "test"
39
+ table: output_table
40
+ mode: truncate_insert
41
+
data/example/postgres-filter-lookup_testing_mysql.yml.liquid ADDED
@@ -0,0 +1,38 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+ in:
5
+ type: mysql
6
+ host: localhost
7
+ user: root
8
+ password: 'passsword'
9
+ database: test
10
+ table: customer
11
+ filters:
12
+ - type: postgress_lookup
13
+ host: localhost
14
+ port: 5432
15
+ database: test
16
+ username: postgres
17
+ password: 1234
18
+ schema: public
19
+ table_name: car
20
+ mapping_from:
21
+ - car_name
22
+ - company
23
+ mapping_to:
24
+ - model
25
+ - brand
26
+ new_columns:
27
+ - { name: car_id, type: string }
28
+ - { name: category, type: string }
29
+ - { name: fuel_capacity, type: string }
30
+ out:
31
+ type: mysql
32
+ host: localhost
33
+ user: root
34
+ password: 'passsword'
35
+ database: test
36
+ table: output_table
37
+ mode: truncate_insert
38
+
data/example/postgres-filter-lookup_testing_oracle.yml.liquid ADDED
@@ -0,0 +1,43 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+ in:
5
+ type: jdbc
6
+ host: localhost
7
+ driver_path: "..path../ojdbc8.jar"
8
+ driver_class: 'oracle.jdbc.driver.OracleDriver'
9
+ url: jdbc:oracle:thin:@localhost:1521:orcl
10
+ user: MYUSER
11
+ password: ABCD
12
+ database: DEMO
13
+ table: customer
14
+ filters:
15
+ - type: postgress_lookup
16
+ host: localhost
17
+ port: 5432
18
+ database: test
19
+ username: postgres
20
+ password: 1234
21
+ schema: public
22
+ table_name: car
23
+ mapping_from:
24
+ - car_name
25
+ - company
26
+ mapping_to:
27
+ - model
28
+ - brand
29
+ new_columns:
30
+ - { name: car_id, type: string }
31
+ - { name: category, type: string }
32
+ - { name: fuel_capacity, type: string }
33
+ out:
34
+ type: jdbc
35
+ host: localhost
36
+ driver_path: "..path../ojdbc8.jar"
37
+ driver_class: 'oracle.jdbc.driver.OracleDriver'
38
+ url: jdbc:oracle:thin:@localhost:1521:orcl
39
+ user: MYUSER
40
+ password: ABCD
41
+ database: DEMO
42
+ table: output_table
43
+ mode: insert_direct
data/src/main/java/org/embulk/filter/postgress_lookup/PostGresConnection.java CHANGED
@@ -28,9 +28,9 @@ public class PostGresConnection {
28
28
  try{
29
29
  // Class.forName("org.postgresql.Driver");
30
30
  if(task.getDriverClass().isPresent()){
31
- this.loadMySqlJdbcDriver(task.getDriverClass().get(),task.getDriverPath());
31
+ this.loadPostgressJdbcDriver(task.getDriverClass().get(),task.getDriverPath());
32
32
  }else{
33
- this.loadMySqlJdbcDriver("org.postgresql.Driver",task.getDriverPath());
33
+ this.loadPostgressJdbcDriver("org.postgresql.Driver",task.getDriverPath());
34
34
  }
35
35
  String url = "jdbc:postgresql://" + task.getHost() + ":"+task.getPort()+"/"+task.getDatabase();
36
36
  connection= DriverManager.getConnection(url, task.getUserName(), task.getPassword());
@@ -58,68 +58,71 @@ public class PostGresConnection {
58
58
 
59
59
  return connection;
60
60
  }
61
- private Class<? extends java.sql.Driver> loadMySqlJdbcDriver(
61
+ private Class<? extends java.sql.Driver> loadPostgressJdbcDriver(
62
62
  final String className,
63
63
  final Optional<String> driverPath)
64
64
  {
65
- synchronized (mysqlJdbcDriver) {
66
- if (mysqlJdbcDriver.get() != null) {
67
- return mysqlJdbcDriver.get();
65
+ synchronized (postgressJdbcDriver) {
66
+ if (postgressJdbcDriver.get() != null) {
67
+ return postgressJdbcDriver.get();
68
+ }
69
+ if (driverPath.isPresent()) {
70
+ logger.info(
71
+ "\"driver_path\" is set to load the Postgress JDBC driver class \"{}\". Adding it to classpath.", className);
72
+ this.addDriverJarToClasspath(driverPath.get());
68
73
  }
69
-
70
74
  try {
71
75
  // If the class is found from the ClassLoader of the plugin, that is prioritized the highest.
72
76
  final Class<? extends java.sql.Driver> found = loadJdbcDriverClassForName(className);
73
- mysqlJdbcDriver.compareAndSet(null, found);
77
+ postgressJdbcDriver.compareAndSet(null, found);
74
78
 
75
79
  if (driverPath.isPresent()) {
76
80
  logger.warn(
77
- "\"driver_path\" is set while the MySQL JDBC driver class \"{}\" is found from the PluginClassLoader."
81
+ "\"driver_path\" is set while the Postgress JDBC driver class \"{}\" is found from the PluginClassLoader."
78
82
  + " \"driver_path\" is ignored.", className);
79
83
  }
80
84
  return found;
81
85
  }
82
86
  catch (final ClassNotFoundException ex) {
83
- // Pass-through once.
84
- }
85
-
86
- if (driverPath.isPresent()) {
87
- logger.info(
88
- "\"driver_path\" is set to load the MySQL JDBC driver class \"{}\". Adding it to classpath.", className);
89
- this.addDriverJarToClasspath(driverPath.get());
87
+ //throw new ConfigException("The MySQL JDBC driver for the class \"" + className + "\" is not found.", ex);
90
88
  }
91
- else {
92
- final File root = this.findPluginRoot();
93
- final File driverLib = new File(root, "default_jdbc_driver");
94
- final File[] files = driverLib.listFiles(new FileFilter() {
95
- @Override
96
- public boolean accept(final File file)
97
- {
98
- return file.isFile() && file.getName().endsWith(".jar");
99
- }
100
- });
101
- if (files == null || files.length == 0) {
102
- throw new ConfigException(new ClassNotFoundException(
103
- "The MySQL JDBC driver for the class \"" + className + "\" is not found"
104
- + " in \"default_jdbc_driver\" (" + root.getAbsolutePath() + ")."));
105
- }
106
- for (final File file : files) {
107
- logger.info(
108
- "The MySQL JDBC driver for the class \"{}\" is expected to be found"
109
- + " in \"default_jdbc_driver\" at {}.", className, file.getAbsolutePath());
110
- this.addDriverJarToClasspath(file.getAbsolutePath());
89
+ final File root = this.findPluginRoot();
90
+ final File driverLib = new File(root, "default_jdbc_driver");
91
+ final File[] files = driverLib.listFiles(new FileFilter() {
92
+ @Override
93
+ public boolean accept(final File file)
94
+ {
95
+ return file.isFile() && file.getName().endsWith(".jar");
111
96
  }
97
+ });
98
+ if (files == null || files.length == 0) {
99
+ throw new ConfigException(new ClassNotFoundException(
100
+ "The Postgress JDBC driver for the class \"" + className + "\" is not found"
101
+ + " in \"default_jdbc_driver\" (" + root.getAbsolutePath() + ")."));
102
+ }
103
+ for (final File file : files) {
104
+ logger.info(
105
+ "The Postgress JDBC driver for the class \"{}\" is expected to be found"
106
+ + " in \"default_jdbc_driver\" at {}.", className, file.getAbsolutePath());
107
+ this.addDriverJarToClasspath(file.getAbsolutePath());
112
108
  }
113
109
 
114
110
  try {
115
- // Retrying to find the class from the ClassLoader of the plugin.
111
+ // If the class is found from the ClassLoader of the plugin, that is prioritized the highest.
116
112
  final Class<? extends java.sql.Driver> found = loadJdbcDriverClassForName(className);
117
- mysqlJdbcDriver.compareAndSet(null, found);
113
+ postgressJdbcDriver.compareAndSet(null, found);
114
+
115
+ if (driverPath.isPresent()) {
116
+ logger.warn(
117
+ "\"driver_path\" is set while the Postgress JDBC driver class \"{}\" is found from the PluginClassLoader."
118
+ + " \"driver_path\" is ignored.", className);
119
+ }
118
120
  return found;
119
121
  }
120
122
  catch (final ClassNotFoundException ex) {
121
- throw new ConfigException("The MySQL JDBC driver for the class \"" + className + "\" is not found.", ex);
123
+ throw new ConfigException("The Postgress JDBC driver for the class \"" + className + "\" is not found.", ex);
122
124
  }
125
+
123
126
  }
124
127
  }
125
128
 
@@ -129,7 +132,7 @@ public class PostGresConnection {
129
132
  return (Class<? extends java.sql.Driver>) Class.forName(className);
130
133
  }
131
134
 
132
- private static final AtomicReference<Class<? extends Driver>> mysqlJdbcDriver = new AtomicReference<>();
135
+ private static final AtomicReference<Class<? extends Driver>> postgressJdbcDriver = new AtomicReference<>();
133
136
 
134
137
  private static final Logger logger = LoggerFactory.getLogger(PostGresConnection.class);
135
138
 
data/src/main/java/org/embulk/filter/postgress_lookup/PostgressLookupFilterPlugin.java CHANGED
@@ -6,6 +6,8 @@ import org.embulk.config.*;
6
6
  import org.embulk.spi.*;
7
7
  import org.embulk.spi.time.Timestamp;
8
8
  import org.embulk.spi.type.Types;
9
+ import org.slf4j.Logger;
10
+ import org.slf4j.LoggerFactory;
9
11
 
10
12
  import java.sql.*;
11
13
  import java.time.Instant;
@@ -13,6 +15,8 @@ import java.util.*;
13
15
 
14
16
  public class PostgressLookupFilterPlugin
15
17
  implements FilterPlugin {
18
+ private static final Logger logger = LoggerFactory.getLogger(PostgressLookupFilterPlugin.class);
19
+
16
20
  public interface PluginTask
17
21
  extends Task {
18
22
 
@@ -25,7 +29,7 @@ public class PostgressLookupFilterPlugin
25
29
  @Config("database")
26
30
  public String getDatabase();
27
31
 
28
- @Config("table")
32
+ @Config("table_name")
29
33
  public String getTableName();
30
34
 
31
35
  @Config("username")
@@ -208,6 +212,9 @@ public class PostgressLookupFilterPlugin
208
212
  columnConfigList.add(columnConfig);
209
213
  }
210
214
 
215
+ Set<String> unmatchedData = new LinkedHashSet<>();
216
+ List<String> keyColumns = task.getMappingFrom();
217
+
211
218
  while (reader.nextRecord()) {
212
219
 
213
220
  int colNum = 0;
@@ -240,6 +247,8 @@ public class PostgressLookupFilterPlugin
240
247
  List<String> matchedData = new ArrayList<>();
241
248
  if (keyValuePair.containsKey(key)) {
242
249
  matchedData = keyValuePair.get(key);
250
+ }else{
251
+ unmatchedData.add(key);
243
252
  }
244
253
 
245
254
  if (matchedData.size() == 0) {
@@ -256,6 +265,19 @@ public class PostgressLookupFilterPlugin
256
265
  builder.addRecord();
257
266
  }
258
267
 
268
+ String info="\n--------------------Unmatched rows.....................\nMapping Key Columns: ";
269
+ for(int i=0;i<keyColumns.size();i++){
270
+ info+= keyColumns.get(i);
271
+ if(i!=keyColumns.size()-1){
272
+ info+=",";
273
+ }
274
+ }
275
+ info+="\n";
276
+ for(String key: unmatchedData){
277
+ info+= key+"\n";
278
+ }
279
+ logger.info(info);
280
+
259
281
  }
260
282
 
261
283
 
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: embulk-filter-postgres_lookup
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.6
4
+ version: 0.1.7
5
5
  platform: ruby
6
6
  authors:
7
7
  - Infoobjects Inc.
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2023-02-10 00:00:00.000000000 Z
11
+ date: 2023-03-03 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
@@ -49,14 +49,17 @@ files:
49
49
  - LICENSE
50
50
  - README.md
51
51
  - build.gradle
52
- - calendar.csv
53
- - classpath/embulk-filter-postgres_lookup-0.1.6.jar
52
+ - classpath/embulk-filter-postgres_lookup-0.1.7.jar
54
53
  - classpath/mysql-connector-java-8.0.19.jar
55
54
  - classpath/postgresql-9.0-801.jdbc4.jar
56
55
  - classpath/protobuf-java-3.6.1.jar
57
56
  - config/checkstyle/checkstyle.xml
58
57
  - config/checkstyle/default.xml
59
- - example/config.yml
58
+ - example/postgres-filter-lookup_testing.yml.liquid
59
+ - example/postgres-filter-lookup_testing_csv.yml.liquid
60
+ - example/postgres-filter-lookup_testing_mssql.yml.liquid
61
+ - example/postgres-filter-lookup_testing_mysql.yml.liquid
62
+ - example/postgres-filter-lookup_testing_oracle.yml.liquid
60
63
  - gradle/wrapper/gradle-wrapper.jar
61
64
  - gradle/wrapper/gradle-wrapper.properties
62
65
  - gradlew
data/calendar.csv DELETED
@@ -1,10 +0,0 @@
1
- dim_calendar_key,year_number,quarter_number,attr_1
2
- -1,0,0
3
- 19900101,1990,1,FirstIndia
4
- 19900102,1990,2,SecondUSA
5
- 19900103,1990,,1000.56
6
- 19900104,1990,4,40_China
7
- 19900105,1990,5,50_Ukraine
8
- 19900106,1990,2,10_USA
9
- 19900107,1990,3,30_UK
10
- 19900109,1990,6,
data/example/config.yml DELETED
@@ -1,48 +0,0 @@
1
- exec:
2
- max_threads: 2
3
- min_output_tasks: 1
4
- in:
5
- type: file
6
- path_prefix: C:\Users\Abhishek Gupta\Desktop\github\embulk-filter-postgres_lookup\calendar.csv
7
- parser:
8
- type: csv
9
- columns:
10
- - { name: dim_calendar_key, type: long }
11
- - { name: year_number, type: long }
12
- - { name: quarter_number, type: long }
13
- - { name: attr_1, type: string }
14
- filters:
15
- - type: postgress_lookup
16
- driver_path: C:/Users/Abhishek Gupta/Desktop/postgresql-9.0-801.jdbc4.jar
17
- driver_class: org.postgresql.Driver
18
- schema_name: testSchema
19
- host: localhost
20
- port: 5432
21
- database: Abhishek
22
- table: country
23
- username: postgres
24
- password: root
25
- mapping_from:
26
- - quarter_number
27
- - attr_1
28
- mapping_to:
29
- - id
30
- - country_address
31
- new_columns:
32
- - { name: country_name, type: string }
33
- - { name: country_address, type: string }
34
- out:
35
- type: file
36
- path_prefix: C:\Users\Abhishek Gupta\Desktop\B\output.csv
37
- file_ext: csv
38
- formatter:
39
- type: csv
40
- delimiter: "\t"
41
- newline: CRLF
42
- newline_in_field: LF
43
- charset: UTF-8
44
- quote_policy: MINIMAL
45
- quote: '"'
46
- escape: "\\"
47
- null_string: "\\N"
48
- default_timezone: 'UTC'