embulk-filter-mysql_lookup 0.1.6 → 0.1.7

Sign up to get free protection for your applications and to get access to all the features.
Files changed (14) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +79 -65
  3. data/build.gradle +1 -1
  4. data/classpath/embulk-filter-mysql_lookup-0.1.7.jar +0 -0
  5. data/example/mysql-filter-lookup_testing.yml.liquid +37 -0
  6. data/example/mysql-filter-lookup_testing_csv.yml.liquid +54 -0
  7. data/example/mysql-filter-lookup_testing_mssql.yml.liquid +39 -0
  8. data/example/mysql-filter-lookup_testing_oracle.yml.liquid +42 -0
  9. data/example/mysql-filter-lookup_testing_postgres.yml.liquid +42 -0
  10. data/src/main/java/org/embulk/filter/mysql_lookup/DatabaseConnection.java +32 -28
  11. data/src/main/java/org/embulk/filter/mysql_lookup/MysqlLookupFilterPlugin.java +21 -4
  12. metadata +8 -4
  13. data/classpath/embulk-filter-mysql_lookup-0.1.6.jar +0 -0
  14. data/example/config1.yml +0 -47
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: b21673bee3d44790b92f529c7e231d0d59d5a840b9333969036cb54692273294
4
- data.tar.gz: 785a6ed85c5ed162116f643a32fa7912fbbfd5df28e47578e8723bf35f77356d
3
+ metadata.gz: 7486a3575cd34e562c8298dee178b17fc06cd2b1bb0446c84465874200a2fd55
4
+ data.tar.gz: 2f32f73302b28d4a6bcc57e3d69aaeb9dab5825d1d66428e00087e73c8db04f2
5
5
  SHA512:
6
- metadata.gz: 4d414ef92f46c1ff72dd0dd38a385d06e0854f722fdfec3442f906e950f5b78626c186ed79d030305bfb208062f11a8f1e81b0138abce7ba8c7e5b2f505c5b8d
7
- data.tar.gz: 87bec41710718ee63d33659e204d7925387277ab83c9405972a33b614649bcb2d2125143112bc1a1a293721ee69b73a345ace1710ff89b5a75bcb6758f6a6b93
6
+ metadata.gz: 15732f29be816f047cac06d2b9eff18466692e6900c0dd9854735314421fa7bb62fdb92db01cf963700575373c30edf4ba0a75e2134815524637fabe578f2af2
7
+ data.tar.gz: 98a5ac748904ee32de5e51afe3f610fde51082e9bb8357c1e070605cdafd85fcbf6493ec72ccdaa28512df02de375964bd008263952d7cff27a7a3e2a5e38f14
data/README.md CHANGED
@@ -30,93 +30,107 @@ An Embulk filter plugin for Lookup Transformation with MySQL database
30
30
  - **Name of column-1**: column name-1 from input file
31
31
  - **Name of column-2**: column name-2 from input file
32
32
  - **new_columns**: (New generated column names) (required)
33
- - **Name-1,Type-1**: Any Name, Type of the name (name: country_name, type: string)
34
- - **Name-2,Type-2**: Any Name, Type of the name (name: country_address, type: string) etc ...
33
+ - **Name-1,Type-1**: Any Name, Type of the name (name: pin, type: string)
34
+ - **Name-2,Type-2**: Any Name, Type of the name (name: gender, type: string)
35
+ - **Name-3,Type-3**: Any Name, Type of the name (name: phone_number, type: string) etc ...
35
36
  ## Example - columns
36
37
  ​
37
- Input1 for table 1 is as follows :-
38
- ​
38
+ Customer.csv for table 1 is as follows :-
39
+
39
40
  ```
40
- year country_code country_name literacy_rate
41
-
42
- 1990 1 India 80%
43
- 1993 2 USA 83%
44
- 1997 3 JAPAN
45
- 1999 4 China 72%
46
- 2000 5 Ukraine 68%
47
- 2002 6 Italy 79%
48
- 2004 7 UK 75%
49
- 2011 8 NULL 42%
41
+ id customer_name address email car_name company
42
+ 1 John Doe 123 Main St, Anytown USA john.doe@example.com Civic Honda
43
+ 2 Jane Smith 456 Elm St, Anytown USA jane.smith@example.com E-Class Mercedes-Benz
44
+ 3 Bob Johnson 789 Oak St, Anytown USA bob.johnson@example.com GLE-Class Mercedes-Benz
45
+ 4 Amanda Hernandez 999 Cedar St, Anytown USA amanda.hernandez@example.com 911 119
46
+ 5 Tom Brown 567 Pine St, Anytown USA tom.brown@example.com C-Class Mercedes-Benz
47
+ 6 Samantha Davis 890 Cedar St, Anytown USA samantha.davis@example.com Civic Honda
48
+ 7 Mike Wilson 1234 Spruce St, Anytown USA mike.wilson@example.com GLE-Class Mercedes-Benz
49
+ 8 Jason Brown 888 Pine St, Anytown USA jason.brown@example.com 911 Porsche
50
+ 9 David Rodriguez 9010 Oak St, Anytown USA david.rodriguez@example.com GLC-Class Mercedes-Benz
51
+ 10 Mark Davis 666 Spruce St, Anytown USA mark.davis@example.com C-Class Mercedes-Benz
52
+ 11 Chris Thompson 222 Cedar St, Anytown USA chris.thompson@example.com Cayenne Porsche
53
+ 12 Linda Young 555 Birch St, Anytown USA linda.young@example.com RAV4
54
+ 13 Kevin Hernandez 444 Maple St, Anytown USA kevin.hernandez@example.com 911 119
50
55
  ```
51
- ​
52
- Input2 for table 2 is as follows :-
53
- ​
56
+
57
+ Car.csv for table 2 is as follows :-
58
+
54
59
  ```
55
- id country_population country_address country_GDP
56
-
57
- 1 11.3 India 1.67
58
- 2 18.2 USA 16.72
59
- 3 30 JAPAN 5.00
60
- 4 4 China 9.33
61
- 5 57 Ukraine 1.08
62
- 6 63 Italy 2.068
63
- 7 17 UK 2.49
64
- 8 28 UAE 1.18
65
-
66
-
67
- Note: country_population is calculated in Billion and country_GDP is calculated in $USD Trillion
60
+ car_id model brand category fuel_capacity
61
+ 87 GLE-Class Mercedes-Benz SUV 80
62
+ 101 Cayenne Porsche SUV 75
63
+ 119 911 Porsche Sports Car 64
64
+ 205 Accord Honda Sedan 56
65
+ 334 Pilot Honda SUV 70
66
+ 434 CR-v Honda SUV 64
67
+ 559 C-Class Mercedes-Benz Sedan 66
68
+ 603 Civic Honda Sedan 42
69
+ 697 E-Class Mercedes-Benz Sedan 72
70
+ 812 GLC-Class Mercedes-Benz Sedan 68
71
+
72
+
68
73
  ```
69
- ​
74
+
70
75
  As shown in yaml below, columns mentioned in mapping_from will be mapped with columns mentioned in mapping_to
71
76
  ie:
72
77
 
73
- ​
74
- country_code : id
75
- country_name : country_address
78
+ car_name : model
79
+ company : brand
76
80
 
77
- After successful mapping an Output.csv file containing the columns mentioned in new_columns will be generated
78
- ​
79
- ​
81
+ After successful mapping an Output.csv file containing the columns mentioned in new_columns will be generated
80
82
 
81
83
  Output File generated :-
82
- ​
84
+
83
85
  ```
84
- year country_code country_name literacy_rate country_GDP country_population
85
-
86
- 1990 1 India 80% 1.67 11.3
87
- 1993 2 USA 83% 16.72 18.2
88
- 1997 3 JAPAN 5.00 30
89
- 1999 4 China 72% 9.33 4
90
- 2000 5 Ukraine 68% 1.08 57
91
- 2002 6 Italy 79% 2.068 63
92
- 2004 7 UK 75% 2.49 17
93
- 2011 8 NULL 42%
86
+ id customer_name address email car_name company car_id category fuel_capacity
87
+ 1 John Doe 123 Main St, Anytown USA john.doe@example.com Civic Honda 603 Sedan 42
88
+ 2 Jane Smith 456 Elm St, Anytown USA jane.smith@example.com E-Class Mercedes-Benz 697 Sedan 72
89
+ 3 Bob Johnson 789 Oak St, Anytown USA bob.johnson@example.com GLE-Class Mercedes-Benz 87 SUV 80
90
+ 4 Amanda Hernandez 999 Cedar St, Anytown USA amanda.hernandez@example.com 911 119 0
91
+ 5 Tom Brown 567 Pine St, Anytown USA tom.brown@example.com C-Class Mercedes-Benz 559 Sedan 66
92
+ 6 Samantha Davis 890 Cedar St, Anytown USA samantha.davis@example.com Civic Honda 603 Sedan 42
93
+ 7 Mike Wilson 1234 Spruce St, Anytown USA mike.wilson@example.com GLE-Class Mercedes-Benz 87 SUV 80
94
+ 8 Jason Brown 888 Pine St, Anytown USA jason.brown@example.com 911 Porsche 119 Sport Car 64
95
+ 9 David Rodriguez 9010 Oak St, Anytown USA david.rodriguez@example.com GLC-Class Mercedes-Benz 812 SUV 68
96
+ 10 Mark Davis 666 Spruce St, Anytown USA mark.davis@example.com C-Class Mercedes-Benz 559 Sedan 66
97
+ 11 Chris Thompson 222 Cedar St, Anytown USA chris.thompson@example.com Cayenne Porsche 101 SUV 75
98
+ 12 Linda Young 555 Birch St, Anytown USA linda.young@example.com RAV4 \N 0
99
+ 13 Kevin Hernandez 444 Maple St, Anytown USA kevin.hernandez@example.com 911 119 0
94
100
  ```
95
101
  ​
96
102
  ​
97
103
  ​
98
104
  ```yaml
99
- - type: mysql_lookup
100
- host: localhost
101
- port: 1433
102
- database: test
103
- tablename: country_details
104
- username: root
105
- password: root
106
- mapping_from:
107
- - country_code
108
- - country_name
109
- mapping_to:
110
- - id
111
- - country_address
112
- new_columns:
113
- - { name: country_GDP, type: string }
114
- - { name: country_population, type: string }
105
+ filters:
106
+ - type: mysql_lookup
107
+ host: localhost
108
+ port: 3306
109
+ database: test
110
+ table_name: car
111
+ username: root
112
+ password: 'passsword'
113
+ mapping_from:
114
+ - car_name
115
+ - company
116
+ mapping_to:
117
+ - model
118
+ - brand
119
+ new_columns:
120
+ - { name: car_id, type: string }
121
+ - { name: category, type: string }
122
+ - { name: fuel_capacity, type: string }
115
123
  ```
116
124
  ​
117
125
  Notes:
118
- 1. mapping_from attribute should be in same order as mentioned in input file.
126
+ 1. mapping_from attribute should be in the same order as mentioned in the input file.
119
127
  ​
128
+ 2. In case with JDBC plugin if any integer column returned as float/decimal then use to cast that column as long as below
129
+ ```
130
+ column_options:
131
+ id: {value_type: long}
132
+ ```
133
+ 3. Matching columns data types must be int,long and String
120
134
  ## Development
121
135
  ​
122
136
  Run example:
data/build.gradle CHANGED
@@ -13,7 +13,7 @@ configurations {
13
13
  provided
14
14
  }
15
15
 
16
- version = "0.1.6"
16
+ version = "0.1.7"
17
17
 
18
18
  sourceCompatibility = 1.8
19
19
  targetCompatibility = 1.8
data/example/mysql-filter-lookup_testing.yml.liquid ADDED
@@ -0,0 +1,37 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+ in:
5
+ type: mysql
6
+ host: localhost
7
+ user: root
8
+ password: 'passsword'
9
+ database: test
10
+ table: customers
11
+ filters:
12
+ - type: mysql_lookup
13
+ host: localhost
14
+ port: 3306
15
+ database: test
16
+ table_name: car
17
+ username: root
18
+ password: 'passsword'
19
+ mapping_from:
20
+ - car_name
21
+ - company
22
+ mapping_to:
23
+ - model
24
+ - brand
25
+ new_columns:
26
+ - { name: car_id, type: string }
27
+ - { name: category, type: string }
28
+ - { name: fuel_capacity, type: string }
29
+
30
+ out:
31
+ type: mysql
32
+ host: localhost
33
+ user: root
34
+ password: 'passsword'
35
+ database: test
36
+ table: output
37
+ mode: truncate_insert
data/example/mysql-filter-lookup_testing_csv.yml.liquid ADDED
@@ -0,0 +1,54 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+ in:
5
+ type: file
6
+ path_prefix: "..path../customer.csv"
7
+ parser:
8
+ charset: UTF-8
9
+ type: csv
10
+ delimiter: ','
11
+ quote: '"'
12
+ header_line: true
13
+ columns:
14
+ - { name: id, type: long }
15
+ - { name: customer_name, type: string }
16
+ - { name: address, type: string }
17
+ - { name: email, type: string }
18
+ - { name: car_name, type: string }
19
+ - { name: company, type: string }
20
+ filters:
21
+ - type: mysql_lookup
22
+ host: localhost
23
+ port: 3306
24
+ database: test
25
+ table_name: car
26
+ username: root
27
+ password: 'passsword'
28
+ mapping_from:
29
+ - car_name
30
+ - company
31
+ mapping_to:
32
+ - model
33
+ - brand
34
+ new_columns:
35
+ - { name: car_id, type: string }
36
+ - { name: category, type: string }
37
+ - { name: fuel_capacity, type: string }
38
+
39
+
40
+ out:
41
+ type: file
42
+ path_prefix: "..path../output.csv"
43
+ file_ext: csv
44
+ formatter:
45
+ type: csv
46
+ delimiter: ","
47
+ newline: CRLF
48
+ newline_in_field: LF
49
+ charset: UTF-8
50
+ quote_policy: MINIMAL
51
+ quote: '"'
52
+ escape: "\\"
53
+ null_string: "\\N"
54
+ default_timezone: 'UTC'
data/example/mysql-filter-lookup_testing_mssql.yml.liquid ADDED
@@ -0,0 +1,39 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+ in:
5
+ type: sqlserver
6
+ host: Localhost
7
+ driver_path: "..path../mssql-jdbc-10.2.0.jre17.jar"
8
+ user: "user"
9
+ password: "password"
10
+ database: "test"
11
+ table: customers
12
+ filters:
13
+ - type: mysql_lookup
14
+ host: localhost
15
+ port: 3306
16
+ database: test
17
+ table_name: car
18
+ username: root
19
+ password: 'passsword'
20
+ mapping_from:
21
+ - car_name
22
+ - company
23
+ mapping_to:
24
+ - model
25
+ - brand
26
+ new_columns:
27
+ - { name: car_id, type: string }
28
+ - { name: category, type: string }
29
+ - { name: fuel_capacity, type: string }
30
+
31
+ out:
32
+ type: sqlserver
33
+ host: Localhost
34
+ driver_path: "..path../mssql-jdbc-10.2.0.jre17.jar"
35
+ user: "usert"
36
+ password: "password"
37
+ database: "test"
38
+ table: output_table
39
+ mode: truncate_insert
data/example/mysql-filter-lookup_testing_oracle.yml.liquid ADDED
@@ -0,0 +1,42 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+ in:
5
+ type: jdbc
6
+ host: localhost
7
+ driver_path: "..path../ojdbc8.jar"
8
+ driver_class: 'oracle.jdbc.driver.OracleDriver'
9
+ url: jdbc:oracle:thin:@localhost:1521:orcl
10
+ user: MYUSER
11
+ password: ABCD
12
+ database: DEMO
13
+ table: customers
14
+ filters:
15
+ - type: mysql_lookup
16
+ host: localhost
17
+ port: 3306
18
+ database: test
19
+ table_name: car
20
+ username: root
21
+ password: 'passsword'
22
+ mapping_from:
23
+ - car_name
24
+ - company
25
+ mapping_to:
26
+ - model
27
+ - brand
28
+ new_columns:
29
+ - { name: car_id, type: string }
30
+ - { name: category, type: string }
31
+ - { name: fuel_capacity, type: string }
32
+ out:
33
+ type: jdbc
34
+ host: localhost
35
+ driver_path: "..path../ojdbc8.jar"
36
+ driver_class: 'oracle.jdbc.driver.OracleDriver'
37
+ url: jdbc:oracle:thin:@localhost:1521:orcl
38
+ user: MYUSER
39
+ password: ABCD
40
+ database: DEMO
41
+ table: output_table
42
+ mode: truncate_insert
data/example/mysql-filter-lookup_testing_postgres.yml.liquid ADDED
@@ -0,0 +1,42 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+ in:
5
+ type: postgresql
6
+ host: localhost
7
+ port: 5432
8
+ user: postgres
9
+ password: 1234
10
+ schema: public
11
+ database: test
12
+ table: customer
13
+ filters:
14
+ - type: mysql_lookup
15
+ host: localhost
16
+ port: 3306
17
+ database: test
18
+ table_name: car
19
+ username: root
20
+ password: 'passsword'
21
+ mapping_from:
22
+ - car_name
23
+ - company
24
+ mapping_to:
25
+ - model
26
+ - brand
27
+ new_columns:
28
+ - { name: car_id, type: string }
29
+ - { name: category, type: string }
30
+ - { name: fuel_capacity, type: string }
31
+
32
+ out:
33
+ type: postgresql
34
+ host: localhost
35
+ port: 5432
36
+ database: test
37
+ user: postgres
38
+ password: 1234
39
+ schema: public
40
+ table: output_table
41
+ mode: truncate_insert
42
+
data/src/main/java/org/embulk/filter/mysql_lookup/DatabaseConnection.java CHANGED
@@ -66,6 +66,11 @@ public class DatabaseConnection {
66
66
  return mysqlJdbcDriver.get();
67
67
  }
68
68
 
69
+ if (driverPath.isPresent()) {
70
+ logger.info(
71
+ "\"driver_path\" is set to load the MySQL JDBC driver class \"{}\". Adding it to classpath.", className);
72
+ this.addDriverJarToClasspath(driverPath.get());
73
+ }
69
74
  try {
70
75
  // If the class is found from the ClassLoader of the plugin, that is prioritized the highest.
71
76
  final Class<? extends java.sql.Driver> found = loadJdbcDriverClassForName(className);
@@ -79,46 +84,45 @@ public class DatabaseConnection {
79
84
  return found;
80
85
  }
81
86
  catch (final ClassNotFoundException ex) {
82
- // Pass-through once.
83
- }
84
-
85
- if (driverPath.isPresent()) {
86
- logger.info(
87
- "\"driver_path\" is set to load the MySQL JDBC driver class \"{}\". Adding it to classpath.", className);
88
- this.addDriverJarToClasspath(driverPath.get());
87
+ //throw new ConfigException("The MySQL JDBC driver for the class \"" + className + "\" is not found.", ex);
89
88
  }
90
- else {
91
- final File root = this.findPluginRoot();
92
- final File driverLib = new File(root, "default_jdbc_driver");
93
- final File[] files = driverLib.listFiles(new FileFilter() {
94
- @Override
95
- public boolean accept(final File file)
96
- {
97
- return file.isFile() && file.getName().endsWith(".jar");
98
- }
99
- });
100
- if (files == null || files.length == 0) {
101
- throw new ConfigException(new ClassNotFoundException(
102
- "The MySQL JDBC driver for the class \"" + className + "\" is not found"
103
- + " in \"default_jdbc_driver\" (" + root.getAbsolutePath() + ")."));
104
- }
105
- for (final File file : files) {
106
- logger.info(
107
- "The MySQL JDBC driver for the class \"{}\" is expected to be found"
108
- + " in \"default_jdbc_driver\" at {}.", className, file.getAbsolutePath());
109
- this.addDriverJarToClasspath(file.getAbsolutePath());
89
+ final File root = this.findPluginRoot();
90
+ final File driverLib = new File(root, "default_jdbc_driver");
91
+ final File[] files = driverLib.listFiles(new FileFilter() {
92
+ @Override
93
+ public boolean accept(final File file)
94
+ {
95
+ return file.isFile() && file.getName().endsWith(".jar");
110
96
  }
97
+ });
98
+ if (files == null || files.length == 0) {
99
+ throw new ConfigException(new ClassNotFoundException(
100
+ "The MySQL JDBC driver for the class \"" + className + "\" is not found"
101
+ + " in \"default_jdbc_driver\" (" + root.getAbsolutePath() + ")."));
102
+ }
103
+ for (final File file : files) {
104
+ logger.info(
105
+ "The MySQL JDBC driver for the class \"{}\" is expected to be found"
106
+ + " in \"default_jdbc_driver\" at {}.", className, file.getAbsolutePath());
107
+ this.addDriverJarToClasspath(file.getAbsolutePath());
111
108
  }
112
109
 
113
110
  try {
114
- // Retrying to find the class from the ClassLoader of the plugin.
111
+ // If the class is found from the ClassLoader of the plugin, that is prioritized the highest.
115
112
  final Class<? extends java.sql.Driver> found = loadJdbcDriverClassForName(className);
116
113
  mysqlJdbcDriver.compareAndSet(null, found);
114
+
115
+ if (driverPath.isPresent()) {
116
+ logger.warn(
117
+ "\"driver_path\" is set while the MySQL JDBC driver class \"{}\" is found from the PluginClassLoader."
118
+ + " \"driver_path\" is ignored.", className);
119
+ }
117
120
  return found;
118
121
  }
119
122
  catch (final ClassNotFoundException ex) {
120
123
  throw new ConfigException("The MySQL JDBC driver for the class \"" + className + "\" is not found.", ex);
121
124
  }
125
+
122
126
  }
123
127
  }
124
128
 
data/src/main/java/org/embulk/filter/mysql_lookup/MysqlLookupFilterPlugin.java CHANGED
@@ -4,15 +4,18 @@ import org.embulk.config.*;
4
4
  import org.embulk.spi.*;
5
5
  import org.embulk.spi.time.Timestamp;
6
6
  import org.embulk.spi.type.Types;
7
-
8
7
  import java.sql.*;
9
8
  import java.time.Instant;
10
9
  import java.util.*;
11
-
10
+ import org.slf4j.Logger;
11
+ import org.slf4j.LoggerFactory;
12
12
  public class MysqlLookupFilterPlugin
13
13
  implements FilterPlugin {
14
+
15
+ private static final Logger logger = LoggerFactory.getLogger(MysqlLookupFilterPlugin.class);
14
16
  public interface PluginTask
15
17
  extends Task {
18
+
16
19
  @Config("host")
17
20
  public String getHost();
18
21
 
@@ -185,7 +188,8 @@ public class MysqlLookupFilterPlugin
185
188
  for (ColumnConfig columnConfig : task.getNewColumns().getColumns()) {
186
189
  columnConfigList.add(columnConfig);
187
190
  }
188
-
191
+ Set<String> unmatchedData = new LinkedHashSet<>();
192
+ List<String> keyColumns = task.getMappingFrom();
189
193
  while (reader.nextRecord()) {
190
194
 
191
195
  int colNum = 0;
@@ -218,6 +222,8 @@ public class MysqlLookupFilterPlugin
218
222
  List<String> matchedData = new ArrayList<>();
219
223
  if (keyValuePair.containsKey(key)) {
220
224
  matchedData = keyValuePair.get(key);
225
+ }else{
226
+ unmatchedData.add(key);
221
227
  }
222
228
 
223
229
  if (matchedData.size() == 0) {
@@ -233,7 +239,18 @@ public class MysqlLookupFilterPlugin
233
239
  }
234
240
  builder.addRecord();
235
241
  }
236
-
242
+ String info="\n--------------------Unmatched rows.....................\nMapping Key Columns: ";
243
+ for(int i=0;i<keyColumns.size();i++){
244
+ info+= keyColumns.get(i);
245
+ if(i!=keyColumns.size()-1){
246
+ info+=",";
247
+ }
248
+ }
249
+ info+="\n";
250
+ for(String key: unmatchedData){
251
+ info+= key+"\n";
252
+ }
253
+ logger.info(info);
237
254
  }
238
255
 
239
256
 
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: embulk-filter-mysql_lookup
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.6
4
+ version: 0.1.7
5
5
  platform: ruby
6
6
  authors:
7
7
  - InfoObjects Inc.
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2023-02-10 00:00:00.000000000 Z
11
+ date: 2023-03-03 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
@@ -49,12 +49,16 @@ files:
49
49
  - LICENSE
50
50
  - README.md
51
51
  - build.gradle
52
- - classpath/embulk-filter-mysql_lookup-0.1.6.jar
52
+ - classpath/embulk-filter-mysql_lookup-0.1.7.jar
53
53
  - classpath/mysql-connector-java-8.0.19.jar
54
54
  - classpath/protobuf-java-3.6.1.jar
55
55
  - config/checkstyle/checkstyle.xml
56
56
  - config/checkstyle/default.xml
57
- - example/config1.yml
57
+ - example/mysql-filter-lookup_testing.yml.liquid
58
+ - example/mysql-filter-lookup_testing_csv.yml.liquid
59
+ - example/mysql-filter-lookup_testing_mssql.yml.liquid
60
+ - example/mysql-filter-lookup_testing_oracle.yml.liquid
61
+ - example/mysql-filter-lookup_testing_postgres.yml.liquid
58
62
  - gradle/wrapper/gradle-wrapper.jar
59
63
  - gradle/wrapper/gradle-wrapper.properties
60
64
  - gradlew
data/example/config1.yml DELETED
@@ -1,47 +0,0 @@
1
- exec:
2
- max_threads: 2
3
- min_output_tasks: 1
4
- in:
5
- type: file
6
- path_prefix: C:\Users\AnkitKumar\Desktop\embulk-filter-mysqllookup\calendar.csv
7
- parser:
8
- type: csv
9
- columns:
10
- - { name: dim_calendar_key, type: long }
11
- - { name: year_number, type: long }
12
- - { name: quarter_number, type: long }
13
- - { name: attr_1, type: string }
14
- filters:
15
- - type: mysql_lookup
16
- host: localhost
17
- port: 3306
18
- driver_path: C:/Users/AnkitKumar/Desktop/embulk-plugins/mysql-connector-java-8.0.19.jar
19
- driver_class: com.mysql.cj.jdbc.Driver
20
- database: information
21
- table_name: country
22
- username: root
23
- password: root
24
- mapping_from:
25
- - quarter_number
26
- - attr_1
27
- mapping_to:
28
- - id
29
- - country_code
30
- new_columns:
31
- - { name: country_address, type: string }
32
- - { name: country_code,type: double}
33
- out:
34
- type: file
35
- path_prefix: C:/Users/AnkitKumar/Desktop/embulk-files/sample/output.csv
36
- file_ext: csv
37
- formatter:
38
- type: csv
39
- delimiter: "\t"
40
- newline: CRLF
41
- newline_in_field: LF
42
- charset: UTF-8
43
- quote_policy: MINIMAL
44
- quote: '"'
45
- escape: "\\"
46
- null_string: "\\N"
47
- default_timezone: 'UTC'