embulk-filter-oracle_lookup 0.2.4 → 0.2.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (17) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +86 -70
  3. data/build.gradle +2 -2
  4. data/classpath/embulk-filter-oracle_lookup-0.2.5.jar +0 -0
  5. data/example/oracle-filter-lookup_testing.yml.liquid +40 -0
  6. data/example/oracle-filter-lookup_testing_csv.yml.liquid +53 -0
  7. data/example/oracle-filter-lookup_testing_mssql.yml.liquid +37 -0
  8. data/example/oracle-filter-lookup_testing_mysql.yml.liquid +34 -0
  9. data/example/oracle-filter-lookup_testing_postgres.yml.liquid +40 -0
  10. data/src/main/java/org/embulk/filter/oracle_lookup/OracleConnection.java +27 -51
  11. data/src/main/java/org/embulk/filter/oracle_lookup/OracleLookupFilterPlugin.java +38 -8
  12. metadata +8 -7
  13. data/calendar.csv +0 -10
  14. data/classpath/embulk-filter-oracle_lookup-0.2.4.jar +0 -0
  15. data/classpath/mysql-connector-java-8.0.19.jar +0 -0
  16. data/classpath/protobuf-java-3.6.1.jar +0 -0
  17. data/example/config.yml +0 -47
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: d7dbfab4e8505300e72dce8c884edad0e537d0f90c662f74c925af89723d1352
4
- data.tar.gz: '030388509dc23a48df7e35c0bb0dbdf58e043b02deb9858728bb1ba17a138b74'
3
+ metadata.gz: a4d201a6b3ae08b5d26da16dd35171de259dec14f4b365eee3e373c23015b8f0
4
+ data.tar.gz: 8385fd851b6e517c2228d4a391d498fff3e4efceca2e0db3db2e69b4767910bc
5
5
  SHA512:
6
- metadata.gz: 0f3e60f4fa8a9496774069d51dc41670dd41e59f8370227b3e163fc5d3c974a603c3a3eb3a31b0462f5c12ba627bc14568a3f88e3da233f0cff5b23f7e33d956
7
- data.tar.gz: 6da66e21c44487ac61a6976795c8d74471686e91c7e36ae24414bd8da4d5ed83e365745931c03b36d2cbc2794eb85558295d7b1370ef13d31803f808f67bfcfc
6
+ metadata.gz: 210a2a22904444a7102a7aa2ede014741b6f6314d0d92474034d9395ad3a1ca3a5f19aef5eeb89b57e4131927f8ddb26120f3403b898955547eb7fe80410c3f2
7
+ data.tar.gz: a89dcb296577116eb391c8fcec467bf7230b4291bc05e89c81bc8c9d88bc58f94c60f9b8ab0c19e252d01312cfec79e42e18ca8f7abec5e11f57bb7785bd11a1
data/README.md CHANGED
@@ -5,22 +5,23 @@
5
5
  <p align="center">Infoobjects is a consulting company that helps enterprises transform how and where they run applications and infrastructure.
6
6
  From strategy, to implementation, to ongoing managed services, Infoobjects creates tailored cloud solutions for enterprises at all stages of the cloud journey.</p>
7
7
 
8
- # Postgres lookup filter plugin for Embulk
8
+ # Oracle lookup filter plugin for Embulk
9
9
  [![license](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
10
10
 
11
- # Oracle lookup filter plugin for Embulk
12
11
  ​
13
12
  An Embulk filter plugin for Lookup Transformation with Oracle database
14
13
  ​
15
14
  ## Configuration
16
15
  ​
17
16
  - **oracle_lookup**: Required attributes for the LookUp Filter Plugin -
18
- - **driver_path**: driver path for your driver which you installed in your system (example `C:/Users/Abhishek Gupta/Desktop/mysql-connector-java-8.0.29-tidb-1.0.1.jar`)
19
- - **driver_class**: driver class name (example `oracle.jdbc.driver.OracleDriver`)
17
+ - **driver_path**: driver path for your driver which you installed in your system(required) (example `....Path..../jarname.jar`)
18
+ - **driver_class**: driver class name(required) (example `oracle.jdbc.driver.OracleDriver`)
20
19
  - **host**: database host (example `localhost`) (required)
21
20
  - **port**: database port (example port for oracle `1521`) (required)
22
21
  - **database**: database name (required)
23
- - **table**: table name of your database (required)
22
+ - **url**: url name (Either url or sid must be provided)
23
+ - **sid_name**: sid name (Either url or sid must be provided)
24
+ - **table_name**: table name of your database (required)
24
25
  - **username**: username for your database (required)
25
26
  - **password**: password for database (required)
26
27
  - **mapping_from**: (Name of columns to be matched with table 2 columns) (required)
@@ -30,93 +31,108 @@ An Embulk filter plugin for Lookup Transformation with Oracle database
30
31
  - **Name of column-1**: column name-1 from input file
31
32
  - **Name of column-2**: column name-2 from input file
32
33
  - **new_columns**: (New generated column names) (required)
33
- - **Name-1,Type-1**: Any Name, Type of the name (name: country_name, type: string)
34
- - **Name-2,Type-2**: Any Name, Type of the name (name: country_address, type: string) etc ...
34
+ - **Name-1,Type-1**: Any Name, Type of the name (name: pin, type: string)
35
+ - **Name-2,Type-2**: Any Name, Type of the name (name: gender, type: string)
36
+ - **Name-3,Type-3**: Any Name, Type of the name (name: phone_number, type: string) etc ...
35
37
  ## Example - columns
36
38
  ​
37
- Input1 for table 1 is as follows :-
38
- ​
39
+ Customer.csv for table 1 is as follows :-
40
+
39
41
  ```
40
- year country_code country_name literacy_rate
41
-
42
- 1990 1 India 80%
43
- 1993 2 USA 83%
44
- 1997 3 JAPAN
45
- 1999 4 China 72%
46
- 2000 5 Ukraine 68%
47
- 2002 6 Italy 79%
48
- 2004 7 UK 75%
49
- 2011 8 NULL 42%
42
+ id customer_name address email car_name company
43
+ 1 John Doe 123 Main St, Anytown USA john.doe@example.com Civic Honda
44
+ 2 Jane Smith 456 Elm St, Anytown USA jane.smith@example.com E-Class Mercedes-Benz
45
+ 3 Bob Johnson 789 Oak St, Anytown USA bob.johnson@example.com GLE-Class Mercedes-Benz
46
+ 4 Amanda Hernandez 999 Cedar St, Anytown USA amanda.hernandez@example.com 911 119
47
+ 5 Tom Brown 567 Pine St, Anytown USA tom.brown@example.com C-Class Mercedes-Benz
48
+ 6 Samantha Davis 890 Cedar St, Anytown USA samantha.davis@example.com Civic Honda
49
+ 7 Mike Wilson 1234 Spruce St, Anytown USA mike.wilson@example.com GLE-Class Mercedes-Benz
50
+ 8 Jason Brown 888 Pine St, Anytown USA jason.brown@example.com 911 Porsche
51
+ 9 David Rodriguez 9010 Oak St, Anytown USA david.rodriguez@example.com GLC-Class Mercedes-Benz
52
+ 10 Mark Davis 666 Spruce St, Anytown USA mark.davis@example.com C-Class Mercedes-Benz
53
+ 11 Chris Thompson 222 Cedar St, Anytown USA chris.thompson@example.com Cayenne Porsche
54
+ 12 Linda Young 555 Birch St, Anytown USA linda.young@example.com RAV4
55
+ 13 Kevin Hernandez 444 Maple St, Anytown USA kevin.hernandez@example.com 911 119
50
56
  ```
51
- ​
52
- Input2 for table 2 is as follows :-
53
- ​
57
+
58
+ Car.csv for table 2 is as follows :-
59
+
54
60
  ```
55
- id country_population country_address country_GDP
56
-
57
- 1 11.3 India 1.67
58
- 2 18.2 USA 16.72
59
- 3 30 JAPAN 5.00
60
- 4 4 China 9.33
61
- 5 57 Ukraine 1.08
62
- 6 63 Italy 2.068
63
- 7 17 UK 2.49
64
- 8 28 UAE 1.18
65
-
66
-
67
- Note: country_population is calculated in Billion and country_GDP is calculated in $USD Trillion
61
+ car_id model brand category fuel_capacity
62
+ 87 GLE-Class Mercedes-Benz SUV 80
63
+ 101 Cayenne Porsche SUV 75
64
+ 119 911 Porsche Sports Car 64
65
+ 205 Accord Honda Sedan 56
66
+ 334 Pilot Honda SUV 70
67
+ 434 CR-v Honda SUV 64
68
+ 559 C-Class Mercedes-Benz Sedan 66
69
+ 603 Civic Honda Sedan 42
70
+ 697 E-Class Mercedes-Benz Sedan 72
71
+ 812 GLC-Class Mercedes-Benz Sedan 68
72
+
73
+
68
74
  ```
69
- ​
75
+
70
76
  As shown in yaml below, columns mentioned in mapping_from will be mapped with columns mentioned in mapping_to
71
77
  ie:
72
78
 
73
- ​
74
- country_code : id
75
- country_name : country_address
79
+ car_name : model
80
+ company : brand
76
81
 
77
- After successful mapping an Output.csv file containing the columns mentioned in new_columns will be generated
78
- ​
79
- ​
82
+ After successful mapping an Output.csv file containing the columns mentioned in new_columns will be generated
80
83
 
81
84
  Output File generated :-
82
- ​
85
+
83
86
  ```
84
- year country_code country_name literacy_rate country_GDP country_population
85
-
86
- 1990 1 India 80% 1.67 11.3
87
- 1993 2 USA 83% 16.72 18.2
88
- 1997 3 JAPAN 5.00 30
89
- 1999 4 China 72% 9.33 4
90
- 2000 5 Ukraine 68% 1.08 57
91
- 2002 6 Italy 79% 2.068 63
92
- 2004 7 UK 75% 2.49 17
93
- 2011 8 NULL 42%
87
+ id customer_name address email car_name company car_id category fuel_capacity
88
+ 1 John Doe 123 Main St, Anytown USA john.doe@example.com Civic Honda 603 Sedan 42
89
+ 2 Jane Smith 456 Elm St, Anytown USA jane.smith@example.com E-Class Mercedes-Benz 697 Sedan 72
90
+ 3 Bob Johnson 789 Oak St, Anytown USA bob.johnson@example.com GLE-Class Mercedes-Benz 87 SUV 80
91
+ 4 Amanda Hernandez 999 Cedar St, Anytown USA amanda.hernandez@example.com 911 119 0
92
+ 5 Tom Brown 567 Pine St, Anytown USA tom.brown@example.com C-Class Mercedes-Benz 559 Sedan 66
93
+ 6 Samantha Davis 890 Cedar St, Anytown USA samantha.davis@example.com Civic Honda 603 Sedan 42
94
+ 7 Mike Wilson 1234 Spruce St, Anytown USA mike.wilson@example.com GLE-Class Mercedes-Benz 87 SUV 80
95
+ 8 Jason Brown 888 Pine St, Anytown USA jason.brown@example.com 911 Porsche 119 Sport Car 64
96
+ 9 David Rodriguez 9010 Oak St, Anytown USA david.rodriguez@example.com GLC-Class Mercedes-Benz 812 SUV 68
97
+ 10 Mark Davis 666 Spruce St, Anytown USA mark.davis@example.com C-Class Mercedes-Benz 559 Sedan 66
98
+ 11 Chris Thompson 222 Cedar St, Anytown USA chris.thompson@example.com Cayenne Porsche 101 SUV 75
99
+ 12 Linda Young 555 Birch St, Anytown USA linda.young@example.com RAV4 \N 0
100
+ 13 Kevin Hernandez 444 Maple St, Anytown USA kevin.hernandez@example.com 911 119 0
101
+
94
102
  ```
95
103
  ​
96
104
  ​
97
105
  ​
98
106
  ```yaml
99
- - type: oracle_lookup
100
- host: localhost
101
- port: 1521
102
- database: XE
103
- tablename: country
104
- username: sys as sysdba
105
- password: root
106
- mapping_from:
107
- - quarter_number
108
- - attr_1
109
- mapping_to:
110
- - id
111
- - country_address
112
- new_columns:
113
- - { name: country_name, type: string }
114
- - { name: country_address, type: string }
107
+ filters:
108
+ - type: oracle_lookup
109
+ url: jdbc:oracle:thin:@localhost:1521:orcl
110
+ driver_path: "C:/drivers/ojdbc8.jar"
111
+ driver_class: 'oracle.jdbc.driver.OracleDriver'
112
+ table_name: car
113
+ username: MYUSER
114
+ password: ABCD
115
+ mapping_from:
116
+ - car_name
117
+ - company
118
+ mapping_to:
119
+ - name_of_car
120
+ - brand
121
+ new_columns:
122
+ - { name: car_id, type: string }
123
+ - { name: category, type: string }
124
+ - { name: fuel_capacity, type: string }
115
125
  ```
116
126
  ​
117
127
  Notes:
118
- 1. mapping_from attribute should be in same order as mentioned in input file.
128
+ 1. mapping_from attribute should be in the same order as mentioned in the input file.
119
129
  ​
130
+ 2. In case with JDBC plugin if any integer column returned as float/decimal then use to cast that column as long as below
131
+ ```
132
+ column_options:
133
+ id: {value_type: long}
134
+ ```
135
+ 3. Matching columns data types must be int,long and String
120
136
  ## Development
121
137
  ​
122
138
  Run example:
data/build.gradle CHANGED
@@ -13,7 +13,7 @@ configurations {
13
13
  provided
14
14
  }
15
15
 
16
- version = "0.2.4"
16
+ version = "0.2.5"
17
17
 
18
18
  sourceCompatibility = 1.8
19
19
  targetCompatibility = 1.8
@@ -23,7 +23,7 @@ dependencies {
23
23
  provided "org.embulk:embulk-core:0.9.8"
24
24
  // compile "YOUR_JAR_DEPENDENCY_GROUP:YOUR_JAR_DEPENDENCY_MODULE:YOUR_JAR_DEPENDENCY_VERSION"
25
25
  testCompile "junit:junit:4.+"
26
- compile group: 'mysql', name: 'mysql-connector-java', version: '8.0.19'
26
+ implementation group: 'com.oracle.database.jdbc', name: 'ojdbc8', version: '21.3.0.0'
27
27
  }
28
28
 
29
29
  task classpath(type: Copy, dependsOn: ["jar"]) {
data/example/oracle-filter-lookup_testing.yml.liquid ADDED
@@ -0,0 +1,40 @@
1
+ in:
2
+ type: jdbc
3
+ driver_path: "C:/drivers/ojdbc8.jar"
4
+ driver_class: 'oracle.jdbc.driver.OracleDriver'
5
+ url: jdbc:oracle:thin:@localhost:1521:orcl
6
+ database: DEMO
7
+ user: MYUSER
8
+ password: ABCD
9
+ table: customers
10
+ column_options:
11
+ id: {value_type: long}
12
+ filters:
13
+ - type: oracle_lookup
14
+ url: jdbc:oracle:thin:@localhost:1521:orcl
15
+ driver_path: "C:/drivers/ojdbc8.jar"
16
+ driver_class: 'oracle.jdbc.driver.OracleDriver'
17
+ table_name: car
18
+ username: MYUSER
19
+ password: ABCD
20
+ mapping_from:
21
+ - car_name
22
+ - company
23
+ mapping_to:
24
+ - name_of_car
25
+ - brand
26
+ new_columns:
27
+ - { name: car_id, type: string }
28
+ - { name: category, type: string }
29
+ - { name: fuel_capacity, type: string }
30
+
31
+ out:
32
+ type: jdbc
33
+ driver_path: "C:/drivers/ojdbc8.jar"
34
+ driver_class: 'oracle.jdbc.driver.OracleDriver'
35
+ url: jdbc:oracle:thin:@localhost:1521:orcl
36
+ database: DEMO
37
+ user: MYUSER
38
+ password: ABCD
39
+ table: output
40
+ mode: insert_direct
data/example/oracle-filter-lookup_testing_csv.yml.liquid ADDED
@@ -0,0 +1,53 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+ in:
5
+ type: file
6
+ path_prefix: "..path../customer.csv"
7
+ parser:
8
+ charset: UTF-8
9
+ type: csv
10
+ delimiter: ','
11
+ quote: '"'
12
+ header_line: true
13
+ columns:
14
+ - { name: id, type: long }
15
+ - { name: customer_name, type: string }
16
+ - { name: address, type: string }
17
+ - { name: email, type: string }
18
+ - { name: car_name, type: string }
19
+ - { name: company, type: string }
20
+ filters:
21
+ - type: oracle_lookup
22
+ url: jdbc:oracle:thin:@localhost:1521:orcl
23
+ driver_path: "C:/drivers/ojdbc8.jar"
24
+ driver_class: 'oracle.jdbc.driver.OracleDriver'
25
+ table_name: car
26
+ username: MYUSER
27
+ password: ABCD
28
+ mapping_from:
29
+ - car_name
30
+ - company
31
+ mapping_to:
32
+ - name_of_car
33
+ - brand
34
+ new_columns:
35
+ - { name: car_id, type: string }
36
+ - { name: category, type: string }
37
+ - { name: fuel_capacity, type: string }
38
+
39
+ out:
40
+ type: file
41
+ path_prefix: "..path../output.csv"
42
+ file_ext: csv
43
+ formatter:
44
+ type: csv
45
+ delimiter: ","
46
+ newline: CRLF
47
+ newline_in_field: LF
48
+ charset: UTF-8
49
+ quote_policy: MINIMAL
50
+ quote: '"'
51
+ escape: "\\"
52
+ null_string: "\\N"
53
+ default_timezone: 'UTC'
data/example/oracle-filter-lookup_testing_mssql.yml.liquid ADDED
@@ -0,0 +1,37 @@
1
+ in:
2
+ type: sqlserver
3
+ host: Localhost
4
+ driver_path: "..path../mssql-jdbc-10.2.0.jre17.jar"
5
+ user: "usert"
6
+ password: "password"
7
+ database: "test"
8
+ table: customers
9
+ filters:
10
+ - type: oracle_lookup
11
+ url: jdbc:oracle:thin:@localhost:1521:orcl
12
+ driver_path: "C:/drivers/ojdbc8.jar"
13
+ driver_class: 'oracle.jdbc.driver.OracleDriver'
14
+ table_name: car
15
+ username: MYUSER
16
+ password: ABCD
17
+ mapping_from:
18
+ - car_name
19
+ - company
20
+ mapping_to:
21
+ - name_of_car
22
+ - brand
23
+ new_columns:
24
+ - { name: car_id, type: string }
25
+ - { name: category, type: string }
26
+ - { name: fuel_capacity, type: string }
27
+
28
+
29
+ out:
30
+ type: sqlserver
31
+ host: Localhost
32
+ driver_path: "..path../mssql-jdbc-10.2.0.jre17.jar"
33
+ user: "usert"
34
+ password: "password"
35
+ database: "test"
36
+ table: output_table
37
+ mode: truncate_insert
data/example/oracle-filter-lookup_testing_mysql.yml.liquid ADDED
@@ -0,0 +1,34 @@
1
+ in:
2
+ type: mysql
3
+ host: localhost
4
+ user: root
5
+ password: 'passsword'
6
+ database: test
7
+ table: customers
8
+ filters:
9
+ - type: oracle_lookup
10
+ url: jdbc:oracle:thin:@localhost:1521:orcl
11
+ driver_path: "C:/drivers/ojdbc8.jar"
12
+ driver_class: 'oracle.jdbc.driver.OracleDriver'
13
+ table_name: car
14
+ username: MYUSER
15
+ password: ABCD
16
+ mapping_from:
17
+ - car_name
18
+ - company
19
+ mapping_to:
20
+ - name_of_car
21
+ - brand
22
+ new_columns:
23
+ - { name: car_id, type: string }
24
+ - { name: category, type: string }
25
+ - { name: fuel_capacity, type: string }
26
+
27
+ out:
28
+ type: mysql
29
+ host: localhost
30
+ user: root
31
+ password: 'passsword'
32
+ database: test
33
+ table: output_table
34
+ mode: truncate_insert
data/example/oracle-filter-lookup_testing_postgres.yml.liquid ADDED
@@ -0,0 +1,40 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+ in:
5
+ type: postgresql
6
+ host: localhost
7
+ port: 5432
8
+ user: postgres
9
+ password: 1234
10
+ schema: public
11
+ database: test
12
+ table: customers
13
+ filters:
14
+ - type: oracle_lookup
15
+ url: jdbc:oracle:thin:@localhost:1521:orcl
16
+ driver_path: "C:/drivers/ojdbc8.jar"
17
+ driver_class: 'oracle.jdbc.driver.OracleDriver'
18
+ table_name: car
19
+ username: MYUSER
20
+ password: ABCD
21
+ mapping_from:
22
+ - car_name
23
+ - company
24
+ mapping_to:
25
+ - name_of_car
26
+ - brand
27
+ new_columns:
28
+ - { name: car_id, type: string }
29
+ - { name: category, type: string }
30
+ - { name: fuel_capacity, type: string }
31
+ out:
32
+ type: postgresql
33
+ host: localhost
34
+ port: 5432
35
+ database: test
36
+ user: postgres
37
+ password: 1234
38
+ schema: public
39
+ table: output_table
40
+ mode: truncate_insert
data/src/main/java/org/embulk/filter/oracle_lookup/OracleConnection.java CHANGED
@@ -25,12 +25,19 @@ public class OracleConnection {
25
25
 
26
26
  private OracleConnection(OracleLookupFilterPlugin.PluginTask task) throws Exception {
27
27
  try{
28
- if(task.getDriverClass().isPresent()){
29
- this.loadMySqlJdbcDriver(task.getDriverClass().get(),task.getDriverPath());
28
+ if(task.getDriverClass().isPresent() && task.getDriverPath().isPresent()){
29
+ this.loadOracleJdbcDriver(task.getDriverClass().get(),task.getDriverPath());
30
30
  }else{
31
- this.loadMySqlJdbcDriver("oracle.jdbc.driver.OracleDriver",task.getDriverPath());
31
+ this.loadOracleJdbcDriver("oracle.jdbc.driver.OracleDriver",task.getDriverPath());
32
+ }
33
+ String url;
34
+ if(task.getURL().isPresent()){
35
+ url= task.getURL().get();
36
+ }else if(task.getSID().isPresent() && task.getHost().isPresent() && task.getPort().isPresent()){
37
+ url="jdbc:oracle:thin:@" + task.getHost().get() + ":"+task.getPort().get()+":"+task.getSID().get();
38
+ }else{
39
+ throw new RuntimeException("sid_name,host and port no must be provided by the user if url has not been provided");
32
40
  }
33
- String url = "jdbc:oracle:thin:@" + task.getHost() + ":"+task.getPort()+":"+task.getDatabase();
34
41
  connection= DriverManager.getConnection(url, task.getUserName(), task.getPassword());
35
42
  }catch (Exception e){
36
43
  e.printStackTrace();
@@ -55,78 +62,47 @@ public class OracleConnection {
55
62
  }
56
63
  return connection;
57
64
  }
58
- private Class<? extends java.sql.Driver> loadMySqlJdbcDriver(
65
+ private Class<? extends java.sql.Driver> loadOracleJdbcDriver(
59
66
  final String className,
60
67
  final Optional<String> driverPath)
61
68
  {
62
- synchronized (mysqlJdbcDriver) {
63
- if (mysqlJdbcDriver.get() != null) {
64
- return mysqlJdbcDriver.get();
69
+ synchronized (oracleJdbcDriver) {
70
+ if (oracleJdbcDriver.get() != null) {
71
+ return oracleJdbcDriver.get();
72
+ }
73
+
74
+ if (driverPath.isPresent()) {
75
+ logger.info(
76
+ "\"driver_path\" is set to load the Oracle JDBC driver class \"{}\". Adding it to classpath.", className);
77
+ this.addDriverJarToClasspath(driverPath.get());
65
78
  }
66
79
 
67
80
  try {
68
81
  // If the class is found from the ClassLoader of the plugin, that is prioritized the highest.
69
- final Class<? extends java.sql.Driver> found = loadJdbcDriverClassForName(className);
70
- mysqlJdbcDriver.compareAndSet(null, found);
82
+ final Class<? extends java.sql.Driver> found = loadOracleJdbcDriverClassForName(className);
83
+ oracleJdbcDriver.compareAndSet(null, found);
71
84
 
72
85
  if (driverPath.isPresent()) {
73
86
  logger.warn(
74
- "\"driver_path\" is set while the MySQL JDBC driver class \"{}\" is found from the PluginClassLoader."
87
+ "\"driver_path\" is set while the Oracle JDBC driver class \"{}\" is found from the PluginClassLoader."
75
88
  + " \"driver_path\" is ignored.", className);
76
89
  }
77
90
  return found;
78
91
  }
79
92
  catch (final ClassNotFoundException ex) {
80
- // Pass-through once.
81
- }
82
-
83
- if (driverPath.isPresent()) {
84
- logger.info(
85
- "\"driver_path\" is set to load the MySQL JDBC driver class \"{}\". Adding it to classpath.", className);
86
- this.addDriverJarToClasspath(driverPath.get());
87
- }
88
- else {
89
- final File root = this.findPluginRoot();
90
- final File driverLib = new File(root, "default_jdbc_driver");
91
- final File[] files = driverLib.listFiles(new FileFilter() {
92
- @Override
93
- public boolean accept(final File file)
94
- {
95
- return file.isFile() && file.getName().endsWith(".jar");
96
- }
97
- });
98
- if (files == null || files.length == 0) {
99
- throw new ConfigException(new ClassNotFoundException(
100
- "The MySQL JDBC driver for the class \"" + className + "\" is not found"
101
- + " in \"default_jdbc_driver\" (" + root.getAbsolutePath() + ")."));
102
- }
103
- for (final File file : files) {
104
- logger.info(
105
- "The MySQL JDBC driver for the class \"{}\" is expected to be found"
106
- + " in \"default_jdbc_driver\" at {}.", className, file.getAbsolutePath());
107
- this.addDriverJarToClasspath(file.getAbsolutePath());
108
- }
93
+ throw new ConfigException("The Oracle JDBC driver for the class \"" + className + "\" is not found.", ex);
109
94
  }
110
95
 
111
- try {
112
- // Retrying to find the class from the ClassLoader of the plugin.
113
- final Class<? extends java.sql.Driver> found = loadJdbcDriverClassForName(className);
114
- mysqlJdbcDriver.compareAndSet(null, found);
115
- return found;
116
- }
117
- catch (final ClassNotFoundException ex) {
118
- throw new ConfigException("The MySQL JDBC driver for the class \"" + className + "\" is not found.", ex);
119
- }
120
96
  }
121
97
  }
122
98
 
123
99
  @SuppressWarnings("unchecked")
124
- private static Class<? extends java.sql.Driver> loadJdbcDriverClassForName(final String className) throws ClassNotFoundException
100
+ private static Class<? extends java.sql.Driver> loadOracleJdbcDriverClassForName(final String className) throws ClassNotFoundException
125
101
  {
126
102
  return (Class<? extends java.sql.Driver>) Class.forName(className);
127
103
  }
128
104
 
129
- private static final AtomicReference<Class<? extends java.sql.Driver>> mysqlJdbcDriver = new AtomicReference<>();
105
+ private static final AtomicReference<Class<? extends java.sql.Driver>> oracleJdbcDriver = new AtomicReference<>();
130
106
 
131
107
  private static final Logger logger = LoggerFactory.getLogger(OracleConnection.class);
132
108
 
data/src/main/java/org/embulk/filter/oracle_lookup/OracleLookupFilterPlugin.java CHANGED
@@ -5,25 +5,29 @@ import org.embulk.config.*;
5
5
  import org.embulk.spi.*;
6
6
  import org.embulk.spi.time.Timestamp;
7
7
  import org.embulk.spi.type.Types;
8
+ import org.slf4j.Logger;
9
+ import org.slf4j.LoggerFactory;
10
+
8
11
  import java.sql.*;
9
12
  import java.time.Instant;
10
13
  import java.util.*;
11
14
 
12
15
  public class OracleLookupFilterPlugin
13
16
  implements FilterPlugin {
17
+ private static final Logger logger = LoggerFactory.getLogger(OracleLookupFilterPlugin.class);
18
+
14
19
  public interface PluginTask
15
20
  extends Task {
16
21
 
17
22
  @Config("host")
18
- public String getHost();
23
+ @ConfigDefault("null")
24
+ public Optional<String> getHost();
19
25
 
20
26
  @Config("port")
21
- public String getPort();
22
-
23
- @Config("database")
24
- public String getDatabase();
27
+ @ConfigDefault("null")
28
+ public Optional<String> getPort();
25
29
 
26
- @Config("table")
30
+ @Config("table_name")
27
31
  public String getTableName();
28
32
 
29
33
  @Config("username")
@@ -48,6 +52,14 @@ public class OracleLookupFilterPlugin
48
52
  @Config("driver_class")
49
53
  @ConfigDefault("null")
50
54
  public Optional<String> getDriverClass();
55
+
56
+ @Config("sid_name")
57
+ @ConfigDefault("null")
58
+ public Optional<String> getSID();
59
+
60
+ @Config("url")
61
+ @ConfigDefault("null")
62
+ public Optional<String> getURL();
51
63
  }
52
64
 
53
65
 
@@ -132,7 +144,7 @@ public class OracleLookupFilterPlugin
132
144
  columnNeedsToBeFetched += newColumns.get(i);
133
145
  }
134
146
  }
135
- query += columnNeedsToBeFetched + " from "+task.getDatabase()+"." +task.getTableName();
147
+ query += columnNeedsToBeFetched + " from "+task.getTableName();
136
148
 
137
149
  Statement stmt = con.createStatement();
138
150
  ResultSet rs = stmt.executeQuery(query);
@@ -200,6 +212,9 @@ public class OracleLookupFilterPlugin
200
212
  columnConfigList.add(columnConfig);
201
213
  }
202
214
 
215
+ Set<String> unmatchedData = new LinkedHashSet<>();
216
+ List<String> keyColumns = task.getMappingFrom();
217
+
203
218
  while (reader.nextRecord()) {
204
219
 
205
220
  int colNum = 0;
@@ -232,6 +247,8 @@ public class OracleLookupFilterPlugin
232
247
  List<String> matchedData = new ArrayList<>();
233
248
  if (keyValuePair.containsKey(key)) {
234
249
  matchedData = keyValuePair.get(key);
250
+ }else{
251
+ unmatchedData.add(key);
235
252
  }
236
253
 
237
254
  if (matchedData.size() == 0) {
@@ -247,6 +264,18 @@ public class OracleLookupFilterPlugin
247
264
  }
248
265
  builder.addRecord();
249
266
  }
267
+ String info="\n--------------------Unmatched rows.....................\nMapping Key Columns: ";
268
+ for(int i=0;i<keyColumns.size();i++){
269
+ info+= keyColumns.get(i);
270
+ if(i!=keyColumns.size()-1){
271
+ info+=",";
272
+ }
273
+ }
274
+ info+="\n";
275
+ for(String key: unmatchedData){
276
+ info+= key+"\n";
277
+ }
278
+ logger.info(info);
250
279
  }
251
280
 
252
281
 
@@ -282,7 +311,8 @@ public class OracleLookupFilterPlugin
282
311
  } else if (Types.DOUBLE.equals(column.getType())) {
283
312
  if (keyMap.get("Key") < inputColumns.size()) {
284
313
  if (column.getName().equalsIgnoreCase(inputColumns.get(keyMap.get("Key")))) {
285
- searchingKeyData.add(String.valueOf(reader.getDouble(column)));
314
+ long num = (long) reader.getDouble(column);
315
+ searchingKeyData.add(String.valueOf(num));
286
316
  int key = keyMap.get("Key");
287
317
  keyMap.put("Key", ++key);
288
318
  }
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: embulk-filter-oracle_lookup
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.4
4
+ version: 0.2.5
5
5
  platform: ruby
6
6
  authors:
7
7
  - Infoobjects Inc.
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2023-02-13 00:00:00.000000000 Z
11
+ date: 2023-03-03 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
@@ -49,13 +49,14 @@ files:
49
49
  - LICENSE
50
50
  - README.md
51
51
  - build.gradle
52
- - calendar.csv
53
- - classpath/embulk-filter-oracle_lookup-0.2.4.jar
54
- - classpath/mysql-connector-java-8.0.19.jar
55
- - classpath/protobuf-java-3.6.1.jar
52
+ - classpath/embulk-filter-oracle_lookup-0.2.5.jar
56
53
  - config/checkstyle/checkstyle.xml
57
54
  - config/checkstyle/default.xml
58
- - example/config.yml
55
+ - example/oracle-filter-lookup_testing.yml.liquid
56
+ - example/oracle-filter-lookup_testing_csv.yml.liquid
57
+ - example/oracle-filter-lookup_testing_mssql.yml.liquid
58
+ - example/oracle-filter-lookup_testing_mysql.yml.liquid
59
+ - example/oracle-filter-lookup_testing_postgres.yml.liquid
59
60
  - gradle/wrapper/gradle-wrapper.jar
60
61
  - gradle/wrapper/gradle-wrapper.properties
61
62
  - gradlew
data/calendar.csv DELETED
@@ -1,10 +0,0 @@
1
- dim_calendar_key,year_number,quarter_number,attr_1
2
- -1,0,0
3
- 19900101,1990,1,FirstIndia
4
- 19900102,1990,2,SecondUSA
5
- 19900103,1990,,1000.56
6
- 19900104,1990,4,40_China
7
- 19900105,1990,5,50_Ukraine
8
- 19900106,1990,2,10_USA
9
- 19900107,1990,3,30_UK
10
- 19900109,1990,7,
data/classpath/protobuf-java-3.6.1.jar DELETED
Binary file
data/example/config.yml DELETED
@@ -1,47 +0,0 @@
1
- exec:
2
- max_threads: 2
3
- min_output_tasks: 1
4
- in:
5
- type: file
6
- path_prefix: C:\Users\Abhishek Gupta\Desktop\github\embulk-filter-oracle_lookup\calendar.csv
7
- parser:
8
- type: csv
9
- columns:
10
- - { name: dim_calendar_key, type: long }
11
- - { name: year_number, type: long }
12
- - { name: quarter_number, type: long }
13
- - { name: attr_1, type: string }
14
- filters:
15
- - type: oracle_lookup
16
- driver_path: C:/Users/Abhishek Gupta/Desktop/mysql-connector-java-8.0.29-tidb-1.0.1.jar
17
- driver_class: oracle.jdbc.driver.OracleDriver
18
- host: localhost
19
- port: 1521
20
- database: XE
21
- table: country
22
- username: sys as sysdba
23
- password: root
24
- mapping_from:
25
- - quarter_number
26
- - attr_1
27
- mapping_to:
28
- - id
29
- - country_address
30
- new_columns:
31
- - { name: country_name, type: string }
32
- - { name: country_address, type: string }
33
- out:
34
- type: file
35
- path_prefix: C:\Users\Abhishek Gupta\Desktop\A\output.csv
36
- file_ext: csv
37
- formatter:
38
- type: csv
39
- delimiter: "\t"
40
- newline: CRLF
41
- newline_in_field: LF
42
- charset: UTF-8
43
- quote_policy: MINIMAL
44
- quote: '"'
45
- escape: "\\"
46
- null_string: "\\N"
47
- default_timezone: 'UTC'