embulk-filter-postgres_lookup 0.1.5 → 0.1.7

Sign up to get free protection for your applications and to get access to all the features.
Files changed (17) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +82 -68
  3. data/build.gradle +1 -1
  4. data/classpath/embulk-filter-postgres_lookup-0.1.7.jar +0 -0
  5. data/example/postgres-filter-lookup_testing.yml.liquid +42 -0
  6. data/example/postgres-filter-lookup_testing_csv.yml.liquid +53 -0
  7. data/example/postgres-filter-lookup_testing_mssql.yml.liquid +41 -0
  8. data/example/postgres-filter-lookup_testing_mysql.yml.liquid +38 -0
  9. data/example/postgres-filter-lookup_testing_oracle.yml.liquid +43 -0
  10. data/gradlew +0 -0
  11. data/gradlew.bat +84 -84
  12. data/src/main/java/org/embulk/filter/postgress_lookup/PostGresConnection.java +197 -194
  13. data/src/main/java/org/embulk/filter/postgress_lookup/PostgressLookupFilterPlugin.java +24 -2
  14. metadata +14 -10
  15. data/calendar.csv +0 -10
  16. data/classpath/embulk-filter-postgres_lookup-0.1.5.jar +0 -0
  17. data/example/config.yml +0 -48
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 2ee540484d77148fd7dd954630c483615bfc8c6e0fb51ee881d806e2567398a3
4
- data.tar.gz: dcfc9d559f7baa9f8ca019717ffc0d79bf33b14745f92079904c6a9fa3b663aa
3
+ metadata.gz: 8b9b5f5cd0836586e6ebf55891c634969983affdfbe4b0cfebd431a1f5262bd1
4
+ data.tar.gz: 7e7984515419c329e7fc86cd99b59578bf6e30f9437880a6a7a179dc2252d861
5
5
  SHA512:
6
- metadata.gz: ce3ae099b68ce81ce44c72efd819ec5a62978f461d92740f5bd34dd865980f53a7787ff1daccc2c1a9661cd14bbadbec79e52ebd3d68222503f481af864e1058
7
- data.tar.gz: 7c792778a130b6613ba8bf954a47eae43be2f4742d851d365693d51d27a3c6e275a803fdf90366042a965393d781bba6f694501d4baad7d76b00c44375e0c7ba
6
+ metadata.gz: 5bf4233c1fc1feef271222e48572560ea143e1a48feabab0838d7375aafd928675bac3bc054898207cf31f6f0e206f34f86beb0ef63184bf946b88f1043687bf
7
+ data.tar.gz: 34a1098823c389d4821e1faabcc56b8d29f5a5b600a093b3f67946490b14871c93a33135497bd68894f23d9191aa79ac4959d5427b0e4e2b817024452e298876
data/README.md CHANGED
@@ -12,13 +12,13 @@ From strategy, to implementation, to ongoing managed services, Infoobjects creat
12
12
  An Embulk filter plugin for Lookup Transformation with Postgres database
13
13
  ​
14
14
  - **postgres_lookup**: Required attributes for the LookUp Filter Plugin -
15
- - **driver_path**: driver path for your driver which you installed in your system (example `C:/Users/Abhishek Gupta/Desktop/postgresql-9.0-801.jdbc4.jar`)
15
+ - **driver_path**: driver path for your driver which you installed in your system (example `....Path..../jarname.jar`)
16
16
  - **driver_class**: driver class name (example `org.postgresql.Driver`)
17
17
  - **schema_name**: schema name (example `testSchema`)
18
18
  - **host**: database host (example `localhost`) (required)
19
19
  - **port**: database port (example port for postgres `1433`) (required)
20
20
  - **database**: database name (required)
21
- - **tablename**: table name of your database (required)
21
+ - **table_name**: table name of your database (required)
22
22
  - **username**: username for your database (required)
23
23
  - **password**: password for database (required)
24
24
  - **mapping_from**: (Name of columns to be matched with table 2 columns) (required)
@@ -28,94 +28,108 @@ An Embulk filter plugin for Lookup Transformation with Postgres database
28
28
  - **Name of column-1**: column name-1 from input file
29
29
  - **Name of column-2**: column name-2 from input file
30
30
  - **new_columns**: (New generated column names) (required)
31
- - **Name-1,Type-1**: Any Name, Type of the name (name: country_name, type: string)
32
- - **Name-2,Type-2**: Any Name, Type of the name (name: country_address, type: string) etc ...
31
+ - **Name-1,Type-1**: Any Name, Type of the name (name: pin, type: string)
32
+ - **Name-2,Type-2**: Any Name, Type of the name (name: gender, type: string)
33
+ - **Name-3,Type-3**: Any Name, Type of the name (name: phone_number, type: string) etc ...
33
34
  ## Example - columns
34
35
  ​
35
- Input1 for table 1 is as follows :-
36
- ​
36
+ Customer.csv for table 1 is as follows :-
37
+
37
38
  ```
38
- year country_code country_name literacy_rate
39
-
40
- 1990 1 India 80%
41
- 1993 2 USA 83%
42
- 1997 3 JAPAN
43
- 1999 4 China 72%
44
- 2000 5 Ukraine 68%
45
- 2002 6 Italy 79%
46
- 2004 7 UK 75%
47
- 2011 8 NULL 42%
39
+ id customer_name address email car_name company
40
+ 1 John Doe 123 Main St, Anytown USA john.doe@example.com Civic Honda
41
+ 2 Jane Smith 456 Elm St, Anytown USA jane.smith@example.com E-Class Mercedes-Benz
42
+ 3 Bob Johnson 789 Oak St, Anytown USA bob.johnson@example.com GLE-Class Mercedes-Benz
43
+ 4 Amanda Hernandez 999 Cedar St, Anytown USA amanda.hernandez@example.com 911 119
44
+ 5 Tom Brown 567 Pine St, Anytown USA tom.brown@example.com C-Class Mercedes-Benz
45
+ 6 Samantha Davis 890 Cedar St, Anytown USA samantha.davis@example.com Civic Honda
46
+ 7 Mike Wilson 1234 Spruce St, Anytown USA mike.wilson@example.com GLE-Class Mercedes-Benz
47
+ 8 Jason Brown 888 Pine St, Anytown USA jason.brown@example.com 911 Porsche
48
+ 9 David Rodriguez 9010 Oak St, Anytown USA david.rodriguez@example.com GLC-Class Mercedes-Benz
49
+ 10 Mark Davis 666 Spruce St, Anytown USA mark.davis@example.com C-Class Mercedes-Benz
50
+ 11 Chris Thompson 222 Cedar St, Anytown USA chris.thompson@example.com Cayenne Porsche
51
+ 12 Linda Young 555 Birch St, Anytown USA linda.young@example.com RAV4
52
+ 13 Kevin Hernandez 444 Maple St, Anytown USA kevin.hernandez@example.com 911 119
48
53
  ```
49
- ​
50
- Input2 for table 2 is as follows :-
51
- ​
54
+
55
+ Car.csv for table 2 is as follows :-
56
+
52
57
  ```
53
- id country_population country_address country_GDP
54
-
55
- 1 11.3 India 1.67
56
- 2 18.2 USA 16.72
57
- 3 30 JAPAN 5.00
58
- 4 4 China 9.33
59
- 5 57 Ukraine 1.08
60
- 6 63 Italy 2.068
61
- 7 17 UK 2.49
62
- 8 28 UAE 1.18
63
-
64
-
65
- Note: country_population is calculated in Billion and country_GDP is calculated in $USD Trillion
58
+ car_id model brand category fuel_capacity
59
+ 87 GLE-Class Mercedes-Benz SUV 80
60
+ 101 Cayenne Porsche SUV 75
61
+ 119 911 Porsche Sports Car 64
62
+ 205 Accord Honda Sedan 56
63
+ 334 Pilot Honda SUV 70
64
+ 434 CR-v Honda SUV 64
65
+ 559 C-Class Mercedes-Benz Sedan 66
66
+ 603 Civic Honda Sedan 42
67
+ 697 E-Class Mercedes-Benz Sedan 72
68
+ 812 GLC-Class Mercedes-Benz Sedan 68
69
+
70
+
66
71
  ```
67
- ​
72
+
68
73
  As shown in yaml below, columns mentioned in mapping_from will be mapped with columns mentioned in mapping_to
69
74
  ie:
70
75
 
71
- ​
72
- country_code : id
73
- country_name : country_address
76
+ car_name : model
77
+ company : brand
74
78
 
75
- After successful mapping an Output.csv file containing the columns mentioned in new_columns will be generated
76
- ​
77
- ​
79
+ After successful mapping an Output.csv file containing the columns mentioned in new_columns will be generated
78
80
 
79
81
  Output File generated :-
80
- ​
82
+
81
83
  ```
82
- year country_code country_name literacy_rate country_GDP country_population
83
-
84
- 1990 1 India 80% 1.67 11.3
85
- 1993 2 USA 83% 16.72 18.2
86
- 1997 3 JAPAN 5.00 30
87
- 1999 4 China 72% 9.33 4
88
- 2000 5 Ukraine 68% 1.08 57
89
- 2002 6 Italy 79% 2.068 63
90
- 2004 7 UK 75% 2.49 17
91
- 2011 8 NULL 42%
84
+ id customer_name address email car_name company car_id category fuel_capacity
85
+ 1 John Doe 123 Main St, Anytown USA john.doe@example.com Civic Honda 603 Sedan 42
86
+ 2 Jane Smith 456 Elm St, Anytown USA jane.smith@example.com E-Class Mercedes-Benz 697 Sedan 72
87
+ 3 Bob Johnson 789 Oak St, Anytown USA bob.johnson@example.com GLE-Class Mercedes-Benz 87 SUV 80
88
+ 4 Amanda Hernandez 999 Cedar St, Anytown USA amanda.hernandez@example.com 911 119 0
89
+ 5 Tom Brown 567 Pine St, Anytown USA tom.brown@example.com C-Class Mercedes-Benz 559 Sedan 66
90
+ 6 Samantha Davis 890 Cedar St, Anytown USA samantha.davis@example.com Civic Honda 603 Sedan 42
91
+ 7 Mike Wilson 1234 Spruce St, Anytown USA mike.wilson@example.com GLE-Class Mercedes-Benz 87 SUV 80
92
+ 8 Jason Brown 888 Pine St, Anytown USA jason.brown@example.com 911 Porsche 119 Sport Car 64
93
+ 9 David Rodriguez 9010 Oak St, Anytown USA david.rodriguez@example.com GLC-Class Mercedes-Benz 812 SUV 68
94
+ 10 Mark Davis 666 Spruce St, Anytown USA mark.davis@example.com C-Class Mercedes-Benz 559 Sedan 66
95
+ 11 Chris Thompson 222 Cedar St, Anytown USA chris.thompson@example.com Cayenne Porsche 101 SUV 75
96
+ 12 Linda Young 555 Birch St, Anytown USA linda.young@example.com RAV4 \N 0
97
+ 13 Kevin Hernandez 444 Maple St, Anytown USA kevin.hernandez@example.com 911 119 0
92
98
  ```
93
99
  ​
94
100
  ​
95
101
  ​
96
102
  ```yaml
97
- - type: postgress_lookup
98
- lookup_destination: mysql
99
- host: localhost
100
- port: 5432
101
- database: Abhishek
102
- tablename: country
103
- username: postgres
104
- password: root
105
- mapping_from:
106
- - quarter_number
107
- - attr_1
108
- mapping_to:
109
- - id
110
- - country_address
111
- new_columns:
112
- - { name: country_name, type: string }
113
- - { name: country_address, type: string }
103
+ filters:
104
+ - type: postgress_lookup
105
+ host: localhost
106
+ port: 5432
107
+ database: test
108
+ username: postgres
109
+ password: 1234
110
+ schema: public
111
+ table_name: car
112
+ mapping_from:
113
+ - car_name
114
+ - company
115
+ mapping_to:
116
+ - model
117
+ - brand
118
+ new_columns:
119
+ - { name: car_id, type: string }
120
+ - { name: category, type: string }
121
+ - { name: fuel_capacity, type: string }
114
122
  ```
115
123
  ​
116
124
  Notes:
117
- 1. mapping_from attribute should be in same order as mentioned in input file.
125
+ 1. mapping_from attribute should be in the same order as mentioned in the input file.
118
126
  ​
127
+ 2. In case with JDBC plugin if any integer column returned as float/decimal then use to cast that column as long as below
128
+ ```
129
+ column_options:
130
+ id: {value_type: long}
131
+ ```
132
+ 3. Matching columns data types must be int,long and String
119
133
  ## Development
120
134
  ​
121
135
  Run example:
data/build.gradle CHANGED
@@ -13,7 +13,7 @@ configurations {
13
13
  provided
14
14
  }
15
15
 
16
- version = "0.1.5"
16
+ version = "0.1.7"
17
17
 
18
18
  sourceCompatibility = 1.8
19
19
  targetCompatibility = 1.8
data/example/postgres-filter-lookup_testing.yml.liquid ADDED
@@ -0,0 +1,42 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+ in:
5
+ type: postgresql
6
+ host: localhost
7
+ port: 5432
8
+ user: postgres
9
+ password: 1234
10
+ schema: public
11
+ database: test
12
+ table: customer
13
+ filters:
14
+ - type: postgress_lookup
15
+ host: localhost
16
+ port: 5432
17
+ database: test
18
+ username: postgres
19
+ password: 1234
20
+ schema: public
21
+ table_name: car
22
+ mapping_from:
23
+ - car_name
24
+ - company
25
+ mapping_to:
26
+ - model
27
+ - brand
28
+ new_columns:
29
+ - { name: car_id, type: string }
30
+ - { name: category, type: string }
31
+ - { name: fuel_capacity, type: string }
32
+ out:
33
+ type: postgresql
34
+ host: localhost
35
+ port: 5432
36
+ database: test
37
+ user: postgres
38
+ password: 1234
39
+ schema: public
40
+ table: output
41
+ mode: truncate_insert
42
+
data/example/postgres-filter-lookup_testing_csv.yml.liquid ADDED
@@ -0,0 +1,53 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+ in:
5
+ type: file
6
+ path_prefix: "..path../customer.csv"
7
+ parser:
8
+ charset: UTF-8
9
+ type: csv
10
+ delimiter: ','
11
+ quote: '"'
12
+ header_line: true
13
+ columns:
14
+ - { name: id, type: long }
15
+ - { name: customer_name, type: string }
16
+ - { name: address, type: string }
17
+ - { name: email, type: string }
18
+ - { name: car_name, type: string }
19
+ - { name: company, type: string }
20
+ filters:
21
+ - type: postgress_lookup
22
+ host: localhost
23
+ port: 5432
24
+ database: test
25
+ username: postgres
26
+ password: 1234
27
+ schema: public
28
+ table_name: car
29
+ mapping_from:
30
+ - car_name
31
+ - company
32
+ mapping_to:
33
+ - model
34
+ - brand
35
+ new_columns:
36
+ - { name: car_id, type: string }
37
+ - { name: category, type: string }
38
+ - { name: fuel_capacity, type: string }
39
+ out:
40
+ type: file
41
+ path_prefix: "..path../output.csv"
42
+ file_ext: csv
43
+ formatter:
44
+ type: csv
45
+ delimiter: ","
46
+ newline: CRLF
47
+ newline_in_field: LF
48
+ charset: UTF-8
49
+ quote_policy: MINIMAL
50
+ quote: '"'
51
+ escape: "\\"
52
+ null_string: "\\N"
53
+ default_timezone: 'UTC'
data/example/postgres-filter-lookup_testing_mssql.yml.liquid ADDED
@@ -0,0 +1,41 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+
5
+ in:
6
+ type: sqlserver
7
+ host: Localhost
8
+ driver_path: "..path../mssql-jdbc-10.2.0.jre17.jar"
9
+ user: "usert"
10
+ password: "password"
11
+ database: "test"
12
+ table: customer
13
+ filters:
14
+ - type: postgress_lookup
15
+ host: localhost
16
+ port: 5432
17
+ database: test
18
+ username: postgres
19
+ password: 1234
20
+ schema: public
21
+ table_name: car
22
+ mapping_from:
23
+ - car_name
24
+ - company
25
+ mapping_to:
26
+ - model
27
+ - brand
28
+ new_columns:
29
+ - { name: car_id, type: string }
30
+ - { name: category, type: string }
31
+ - { name: fuel_capacity, type: string }
32
+ out:
33
+ type: sqlserver
34
+ host: Localhost
35
+ driver_path: "..path../mssql-jdbc-10.2.0.jre17.jar"
36
+ user: "usert"
37
+ password: "password"
38
+ database: "test"
39
+ table: output_table
40
+ mode: truncate_insert
41
+
data/example/postgres-filter-lookup_testing_mysql.yml.liquid ADDED
@@ -0,0 +1,38 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+ in:
5
+ type: mysql
6
+ host: localhost
7
+ user: root
8
+ password: 'passsword'
9
+ database: test
10
+ table: customer
11
+ filters:
12
+ - type: postgress_lookup
13
+ host: localhost
14
+ port: 5432
15
+ database: test
16
+ username: postgres
17
+ password: 1234
18
+ schema: public
19
+ table_name: car
20
+ mapping_from:
21
+ - car_name
22
+ - company
23
+ mapping_to:
24
+ - model
25
+ - brand
26
+ new_columns:
27
+ - { name: car_id, type: string }
28
+ - { name: category, type: string }
29
+ - { name: fuel_capacity, type: string }
30
+ out:
31
+ type: mysql
32
+ host: localhost
33
+ user: root
34
+ password: 'passsword'
35
+ database: test
36
+ table: output_table
37
+ mode: truncate_insert
38
+
data/example/postgres-filter-lookup_testing_oracle.yml.liquid ADDED
@@ -0,0 +1,43 @@
1
+ exec:
2
+ max_threads: 1
3
+ min_output_tasks: 1
4
+ in:
5
+ type: jdbc
6
+ host: localhost
7
+ driver_path: "..path../ojdbc8.jar"
8
+ driver_class: 'oracle.jdbc.driver.OracleDriver'
9
+ url: jdbc:oracle:thin:@localhost:1521:orcl
10
+ user: MYUSER
11
+ password: ABCD
12
+ database: DEMO
13
+ table: customer
14
+ filters:
15
+ - type: postgress_lookup
16
+ host: localhost
17
+ port: 5432
18
+ database: test
19
+ username: postgres
20
+ password: 1234
21
+ schema: public
22
+ table_name: car
23
+ mapping_from:
24
+ - car_name
25
+ - company
26
+ mapping_to:
27
+ - model
28
+ - brand
29
+ new_columns:
30
+ - { name: car_id, type: string }
31
+ - { name: category, type: string }
32
+ - { name: fuel_capacity, type: string }
33
+ out:
34
+ type: jdbc
35
+ host: localhost
36
+ driver_path: "..path../ojdbc8.jar"
37
+ driver_class: 'oracle.jdbc.driver.OracleDriver'
38
+ url: jdbc:oracle:thin:@localhost:1521:orcl
39
+ user: MYUSER
40
+ password: ABCD
41
+ database: DEMO
42
+ table: output_table
43
+ mode: insert_direct
data/gradlew CHANGED
File without changes
data/gradlew.bat CHANGED
@@ -1,84 +1,84 @@
1
- @if "%DEBUG%" == "" @echo off
2
- @rem ##########################################################################
3
- @rem
4
- @rem Gradle startup script for Windows
5
- @rem
6
- @rem ##########################################################################
7
-
8
- @rem Set local scope for the variables with windows NT shell
9
- if "%OS%"=="Windows_NT" setlocal
10
-
11
- set DIRNAME=%~dp0
12
- if "%DIRNAME%" == "" set DIRNAME=.
13
- set APP_BASE_NAME=%~n0
14
- set APP_HOME=%DIRNAME%
15
-
16
- @rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
17
- set DEFAULT_JVM_OPTS=
18
-
19
- @rem Find java.exe
20
- if defined JAVA_HOME goto findJavaFromJavaHome
21
-
22
- set JAVA_EXE=java.exe
23
- %JAVA_EXE% -version >NUL 2>&1
24
- if "%ERRORLEVEL%" == "0" goto init
25
-
26
- echo.
27
- echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
28
- echo.
29
- echo Please set the JAVA_HOME variable in your environment to match the
30
- echo location of your Java installation.
31
-
32
- goto fail
33
-
34
- :findJavaFromJavaHome
35
- set JAVA_HOME=%JAVA_HOME:"=%
36
- set JAVA_EXE=%JAVA_HOME%/bin/java.exe
37
-
38
- if exist "%JAVA_EXE%" goto init
39
-
40
- echo.
41
- echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME%
42
- echo.
43
- echo Please set the JAVA_HOME variable in your environment to match the
44
- echo location of your Java installation.
45
-
46
- goto fail
47
-
48
- :init
49
- @rem Get command-line arguments, handling Windows variants
50
-
51
- if not "%OS%" == "Windows_NT" goto win9xME_args
52
-
53
- :win9xME_args
54
- @rem Slurp the command line arguments.
55
- set CMD_LINE_ARGS=
56
- set _SKIP=2
57
-
58
- :win9xME_args_slurp
59
- if "x%~1" == "x" goto execute
60
-
61
- set CMD_LINE_ARGS=%*
62
-
63
- :execute
64
- @rem Setup the command line
65
-
66
- set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar
67
-
68
- @rem Execute Gradle
69
- "%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %CMD_LINE_ARGS%
70
-
71
- :end
72
- @rem End local scope for the variables with windows NT shell
73
- if "%ERRORLEVEL%"=="0" goto mainEnd
74
-
75
- :fail
76
- rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of
77
- rem the _cmd.exe /c_ return code!
78
- if not "" == "%GRADLE_EXIT_CONSOLE%" exit 1
79
- exit /b 1
80
-
81
- :mainEnd
82
- if "%OS%"=="Windows_NT" endlocal
83
-
84
- :omega
1
+ @if "%DEBUG%" == "" @echo off
2
+ @rem ##########################################################################
3
+ @rem
4
+ @rem Gradle startup script for Windows
5
+ @rem
6
+ @rem ##########################################################################
7
+
8
+ @rem Set local scope for the variables with windows NT shell
9
+ if "%OS%"=="Windows_NT" setlocal
10
+
11
+ set DIRNAME=%~dp0
12
+ if "%DIRNAME%" == "" set DIRNAME=.
13
+ set APP_BASE_NAME=%~n0
14
+ set APP_HOME=%DIRNAME%
15
+
16
+ @rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
17
+ set DEFAULT_JVM_OPTS=
18
+
19
+ @rem Find java.exe
20
+ if defined JAVA_HOME goto findJavaFromJavaHome
21
+
22
+ set JAVA_EXE=java.exe
23
+ %JAVA_EXE% -version >NUL 2>&1
24
+ if "%ERRORLEVEL%" == "0" goto init
25
+
26
+ echo.
27
+ echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
28
+ echo.
29
+ echo Please set the JAVA_HOME variable in your environment to match the
30
+ echo location of your Java installation.
31
+
32
+ goto fail
33
+
34
+ :findJavaFromJavaHome
35
+ set JAVA_HOME=%JAVA_HOME:"=%
36
+ set JAVA_EXE=%JAVA_HOME%/bin/java.exe
37
+
38
+ if exist "%JAVA_EXE%" goto init
39
+
40
+ echo.
41
+ echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME%
42
+ echo.
43
+ echo Please set the JAVA_HOME variable in your environment to match the
44
+ echo location of your Java installation.
45
+
46
+ goto fail
47
+
48
+ :init
49
+ @rem Get command-line arguments, handling Windows variants
50
+
51
+ if not "%OS%" == "Windows_NT" goto win9xME_args
52
+
53
+ :win9xME_args
54
+ @rem Slurp the command line arguments.
55
+ set CMD_LINE_ARGS=
56
+ set _SKIP=2
57
+
58
+ :win9xME_args_slurp
59
+ if "x%~1" == "x" goto execute
60
+
61
+ set CMD_LINE_ARGS=%*
62
+
63
+ :execute
64
+ @rem Setup the command line
65
+
66
+ set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar
67
+
68
+ @rem Execute Gradle
69
+ "%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %CMD_LINE_ARGS%
70
+
71
+ :end
72
+ @rem End local scope for the variables with windows NT shell
73
+ if "%ERRORLEVEL%"=="0" goto mainEnd
74
+
75
+ :fail
76
+ rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of
77
+ rem the _cmd.exe /c_ return code!
78
+ if not "" == "%GRADLE_EXIT_CONSOLE%" exit 1
79
+ exit /b 1
80
+
81
+ :mainEnd
82
+ if "%OS%"=="Windows_NT" endlocal
83
+
84
+ :omega
data/src/main/java/org/embulk/filter/postgress_lookup/PostGresConnection.java CHANGED
@@ -1,194 +1,197 @@
1
- package org.embulk.filter.postgress_lookup;
2
-
3
- import org.embulk.config.ConfigException;
4
- import org.slf4j.Logger;
5
- import org.slf4j.LoggerFactory;
6
-
7
- import java.io.File;
8
- import java.io.FileFilter;
9
- import java.lang.reflect.InvocationTargetException;
10
- import java.lang.reflect.Method;
11
- import java.net.MalformedURLException;
12
- import java.net.URISyntaxException;
13
- import java.net.URL;
14
- import java.net.URLClassLoader;
15
- import java.nio.file.Path;
16
- import java.nio.file.Paths;
17
- import java.sql.Connection;
18
- import java.sql.Driver;
19
- import java.sql.DriverManager;
20
- import java.util.Optional;
21
- import java.util.concurrent.atomic.AtomicReference;
22
-
23
- public class PostGresConnection {
24
-
25
- private static Connection connection=null;
26
-
27
- private PostGresConnection(PostgressLookupFilterPlugin.PluginTask task) throws Exception {
28
- try{
29
- // Class.forName("org.postgresql.Driver");
30
- if(task.getDriverClass().isPresent()){
31
- this.loadMySqlJdbcDriver(task.getDriverClass().get(),task.getDriverPath());
32
- }else{
33
- this.loadMySqlJdbcDriver("org.postgresql.Driver",task.getDriverPath());
34
- }
35
- String url = "jdbc:postgresql://" + task.getHost() + ":"+task.getPort()+"/"+task.getDatabase();
36
- connection= DriverManager.getConnection(url, task.getUserName(), task.getPassword());
37
- }catch (Exception e){
38
- e.printStackTrace();
39
- throw new Exception(e);
40
- }
41
-
42
- }
43
-
44
- public static Connection getConnection(PostgressLookupFilterPlugin.PluginTask task){
45
- try {
46
- if(connection==null || connection.isClosed()){
47
- try {
48
- new PostGresConnection(task);
49
- return connection;
50
- } catch (Exception e) {
51
- e.printStackTrace();
52
- throw new RuntimeException();
53
- }
54
- }
55
- }catch (Exception e){
56
- throw new RuntimeException(e);
57
- }
58
-
59
- return connection;
60
- }
61
- private Class<? extends java.sql.Driver> loadMySqlJdbcDriver(
62
- final String className,
63
- final Optional<String> driverPath)
64
- {
65
- synchronized (mysqlJdbcDriver) {
66
- if (mysqlJdbcDriver.get() != null) {
67
- return mysqlJdbcDriver.get();
68
- }
69
-
70
- try {
71
- // If the class is found from the ClassLoader of the plugin, that is prioritized the highest.
72
- final Class<? extends java.sql.Driver> found = loadJdbcDriverClassForName(className);
73
- mysqlJdbcDriver.compareAndSet(null, found);
74
-
75
- if (driverPath.isPresent()) {
76
- logger.warn(
77
- "\"driver_path\" is set while the MySQL JDBC driver class \"{}\" is found from the PluginClassLoader."
78
- + " \"driver_path\" is ignored.", className);
79
- }
80
- return found;
81
- }
82
- catch (final ClassNotFoundException ex) {
83
- // Pass-through once.
84
- }
85
-
86
- if (driverPath.isPresent()) {
87
- logger.info(
88
- "\"driver_path\" is set to load the MySQL JDBC driver class \"{}\". Adding it to classpath.", className);
89
- this.addDriverJarToClasspath(driverPath.get());
90
- }
91
- else {
92
- final File root = this.findPluginRoot();
93
- final File driverLib = new File(root, "default_jdbc_driver");
94
- final File[] files = driverLib.listFiles(new FileFilter() {
95
- @Override
96
- public boolean accept(final File file)
97
- {
98
- return file.isFile() && file.getName().endsWith(".jar");
99
- }
100
- });
101
- if (files == null || files.length == 0) {
102
- throw new ConfigException(new ClassNotFoundException(
103
- "The MySQL JDBC driver for the class \"" + className + "\" is not found"
104
- + " in \"default_jdbc_driver\" (" + root.getAbsolutePath() + ")."));
105
- }
106
- for (final File file : files) {
107
- logger.info(
108
- "The MySQL JDBC driver for the class \"{}\" is expected to be found"
109
- + " in \"default_jdbc_driver\" at {}.", className, file.getAbsolutePath());
110
- this.addDriverJarToClasspath(file.getAbsolutePath());
111
- }
112
- }
113
-
114
- try {
115
- // Retrying to find the class from the ClassLoader of the plugin.
116
- final Class<? extends java.sql.Driver> found = loadJdbcDriverClassForName(className);
117
- mysqlJdbcDriver.compareAndSet(null, found);
118
- return found;
119
- }
120
- catch (final ClassNotFoundException ex) {
121
- throw new ConfigException("The MySQL JDBC driver for the class \"" + className + "\" is not found.", ex);
122
- }
123
- }
124
- }
125
-
126
- @SuppressWarnings("unchecked")
127
- private static Class<? extends java.sql.Driver> loadJdbcDriverClassForName(final String className) throws ClassNotFoundException
128
- {
129
- return (Class<? extends java.sql.Driver>) Class.forName(className);
130
- }
131
-
132
- private static final AtomicReference<Class<? extends Driver>> mysqlJdbcDriver = new AtomicReference<>();
133
-
134
- private static final Logger logger = LoggerFactory.getLogger(PostGresConnection.class);
135
-
136
- protected void addDriverJarToClasspath(String glob)
137
- {
138
- // TODO match glob
139
- final ClassLoader loader = getClass().getClassLoader();
140
- if (!(loader instanceof URLClassLoader)) {
141
- throw new RuntimeException("Plugin is not loaded by URLClassLoader unexpectedly.");
142
- }
143
- if (!"org.embulk.plugin.PluginClassLoader".equals(loader.getClass().getName())) {
144
- throw new RuntimeException("Plugin is not loaded by PluginClassLoader unexpectedly.");
145
- }
146
- Path path = Paths.get(glob);
147
- if (!path.toFile().exists()) {
148
- throw new ConfigException("The specified driver jar doesn't exist: " + glob);
149
- }
150
- final Method addPathMethod;
151
- try {
152
- addPathMethod = loader.getClass().getMethod("addPath", Path.class);
153
- } catch (final NoSuchMethodException ex) {
154
- throw new RuntimeException("Plugin is not loaded a ClassLoader which has addPath(Path), unexpectedly.");
155
- }
156
- try {
157
- addPathMethod.invoke(loader, Paths.get(glob));
158
- } catch (final IllegalAccessException ex) {
159
- throw new RuntimeException(ex);
160
- } catch (final InvocationTargetException ex) {
161
- final Throwable targetException = ex.getTargetException();
162
- if (targetException instanceof MalformedURLException) {
163
- throw new IllegalArgumentException(targetException);
164
- } else if (targetException instanceof RuntimeException) {
165
- throw (RuntimeException) targetException;
166
- } else {
167
- throw new RuntimeException(targetException);
168
- }
169
- }
170
- }
171
-
172
- protected File findPluginRoot()
173
- {
174
- try {
175
- URL url = getClass().getResource("/" + getClass().getName().replace('.', '/') + ".class");
176
- if (url.toString().startsWith("jar:")) {
177
- url = new URL(url.toString().replaceAll("^jar:", "").replaceAll("![^!]*$", ""));
178
- }
179
-
180
- File folder = new File(url.toURI()).getParentFile();
181
- for (;; folder = folder.getParentFile()) {
182
- if (folder == null) {
183
- throw new RuntimeException("Cannot find 'embulk-filter-xxx' folder.");
184
- }
185
-
186
- if (folder.getName().startsWith("embulk-input-")) {
187
- return folder;
188
- }
189
- }
190
- } catch (URISyntaxException | MalformedURLException e) {
191
- throw new RuntimeException(e);
192
- }
193
- }
194
- }
1
+ package org.embulk.filter.postgress_lookup;
2
+
3
+ import org.embulk.config.ConfigException;
4
+ import org.slf4j.Logger;
5
+ import org.slf4j.LoggerFactory;
6
+
7
+ import java.io.File;
8
+ import java.io.FileFilter;
9
+ import java.lang.reflect.InvocationTargetException;
10
+ import java.lang.reflect.Method;
11
+ import java.net.MalformedURLException;
12
+ import java.net.URISyntaxException;
13
+ import java.net.URL;
14
+ import java.net.URLClassLoader;
15
+ import java.nio.file.Path;
16
+ import java.nio.file.Paths;
17
+ import java.sql.Connection;
18
+ import java.sql.Driver;
19
+ import java.sql.DriverManager;
20
+ import java.util.Optional;
21
+ import java.util.concurrent.atomic.AtomicReference;
22
+
23
+ public class PostGresConnection {
24
+
25
+ private static Connection connection=null;
26
+
27
+ private PostGresConnection(PostgressLookupFilterPlugin.PluginTask task) throws Exception {
28
+ try{
29
+ // Class.forName("org.postgresql.Driver");
30
+ if(task.getDriverClass().isPresent()){
31
+ this.loadPostgressJdbcDriver(task.getDriverClass().get(),task.getDriverPath());
32
+ }else{
33
+ this.loadPostgressJdbcDriver("org.postgresql.Driver",task.getDriverPath());
34
+ }
35
+ String url = "jdbc:postgresql://" + task.getHost() + ":"+task.getPort()+"/"+task.getDatabase();
36
+ connection= DriverManager.getConnection(url, task.getUserName(), task.getPassword());
37
+ }catch (Exception e){
38
+ e.printStackTrace();
39
+ throw new Exception(e);
40
+ }
41
+
42
+ }
43
+
44
+ public static Connection getConnection(PostgressLookupFilterPlugin.PluginTask task){
45
+ try {
46
+ if(connection==null || connection.isClosed()){
47
+ try {
48
+ new PostGresConnection(task);
49
+ return connection;
50
+ } catch (Exception e) {
51
+ e.printStackTrace();
52
+ throw new RuntimeException();
53
+ }
54
+ }
55
+ }catch (Exception e){
56
+ throw new RuntimeException(e);
57
+ }
58
+
59
+ return connection;
60
+ }
61
+ private Class<? extends java.sql.Driver> loadPostgressJdbcDriver(
62
+ final String className,
63
+ final Optional<String> driverPath)
64
+ {
65
+ synchronized (postgressJdbcDriver) {
66
+ if (postgressJdbcDriver.get() != null) {
67
+ return postgressJdbcDriver.get();
68
+ }
69
+ if (driverPath.isPresent()) {
70
+ logger.info(
71
+ "\"driver_path\" is set to load the Postgress JDBC driver class \"{}\". Adding it to classpath.", className);
72
+ this.addDriverJarToClasspath(driverPath.get());
73
+ }
74
+ try {
75
+ // If the class is found from the ClassLoader of the plugin, that is prioritized the highest.
76
+ final Class<? extends java.sql.Driver> found = loadJdbcDriverClassForName(className);
77
+ postgressJdbcDriver.compareAndSet(null, found);
78
+
79
+ if (driverPath.isPresent()) {
80
+ logger.warn(
81
+ "\"driver_path\" is set while the Postgress JDBC driver class \"{}\" is found from the PluginClassLoader."
82
+ + " \"driver_path\" is ignored.", className);
83
+ }
84
+ return found;
85
+ }
86
+ catch (final ClassNotFoundException ex) {
87
+ //throw new ConfigException("The MySQL JDBC driver for the class \"" + className + "\" is not found.", ex);
88
+ }
89
+ final File root = this.findPluginRoot();
90
+ final File driverLib = new File(root, "default_jdbc_driver");
91
+ final File[] files = driverLib.listFiles(new FileFilter() {
92
+ @Override
93
+ public boolean accept(final File file)
94
+ {
95
+ return file.isFile() && file.getName().endsWith(".jar");
96
+ }
97
+ });
98
+ if (files == null || files.length == 0) {
99
+ throw new ConfigException(new ClassNotFoundException(
100
+ "The Postgress JDBC driver for the class \"" + className + "\" is not found"
101
+ + " in \"default_jdbc_driver\" (" + root.getAbsolutePath() + ")."));
102
+ }
103
+ for (final File file : files) {
104
+ logger.info(
105
+ "The Postgress JDBC driver for the class \"{}\" is expected to be found"
106
+ + " in \"default_jdbc_driver\" at {}.", className, file.getAbsolutePath());
107
+ this.addDriverJarToClasspath(file.getAbsolutePath());
108
+ }
109
+
110
+ try {
111
+ // If the class is found from the ClassLoader of the plugin, that is prioritized the highest.
112
+ final Class<? extends java.sql.Driver> found = loadJdbcDriverClassForName(className);
113
+ postgressJdbcDriver.compareAndSet(null, found);
114
+
115
+ if (driverPath.isPresent()) {
116
+ logger.warn(
117
+ "\"driver_path\" is set while the Postgress JDBC driver class \"{}\" is found from the PluginClassLoader."
118
+ + " \"driver_path\" is ignored.", className);
119
+ }
120
+ return found;
121
+ }
122
+ catch (final ClassNotFoundException ex) {
123
+ throw new ConfigException("The Postgress JDBC driver for the class \"" + className + "\" is not found.", ex);
124
+ }
125
+
126
+ }
127
+ }
128
+
129
+ @SuppressWarnings("unchecked")
130
+ private static Class<? extends java.sql.Driver> loadJdbcDriverClassForName(final String className) throws ClassNotFoundException
131
+ {
132
+ return (Class<? extends java.sql.Driver>) Class.forName(className);
133
+ }
134
+
135
+ private static final AtomicReference<Class<? extends Driver>> postgressJdbcDriver = new AtomicReference<>();
136
+
137
+ private static final Logger logger = LoggerFactory.getLogger(PostGresConnection.class);
138
+
139
+ protected void addDriverJarToClasspath(String glob)
140
+ {
141
+ // TODO match glob
142
+ final ClassLoader loader = getClass().getClassLoader();
143
+ if (!(loader instanceof URLClassLoader)) {
144
+ throw new RuntimeException("Plugin is not loaded by URLClassLoader unexpectedly.");
145
+ }
146
+ if (!"org.embulk.plugin.PluginClassLoader".equals(loader.getClass().getName())) {
147
+ throw new RuntimeException("Plugin is not loaded by PluginClassLoader unexpectedly.");
148
+ }
149
+ Path path = Paths.get(glob);
150
+ if (!path.toFile().exists()) {
151
+ throw new ConfigException("The specified driver jar doesn't exist: " + glob);
152
+ }
153
+ final Method addPathMethod;
154
+ try {
155
+ addPathMethod = loader.getClass().getMethod("addPath", Path.class);
156
+ } catch (final NoSuchMethodException ex) {
157
+ throw new RuntimeException("Plugin is not loaded a ClassLoader which has addPath(Path), unexpectedly.");
158
+ }
159
+ try {
160
+ addPathMethod.invoke(loader, Paths.get(glob));
161
+ } catch (final IllegalAccessException ex) {
162
+ throw new RuntimeException(ex);
163
+ } catch (final InvocationTargetException ex) {
164
+ final Throwable targetException = ex.getTargetException();
165
+ if (targetException instanceof MalformedURLException) {
166
+ throw new IllegalArgumentException(targetException);
167
+ } else if (targetException instanceof RuntimeException) {
168
+ throw (RuntimeException) targetException;
169
+ } else {
170
+ throw new RuntimeException(targetException);
171
+ }
172
+ }
173
+ }
174
+
175
+ protected File findPluginRoot()
176
+ {
177
+ try {
178
+ URL url = getClass().getResource("/" + getClass().getName().replace('.', '/') + ".class");
179
+ if (url.toString().startsWith("jar:")) {
180
+ url = new URL(url.toString().replaceAll("^jar:", "").replaceAll("![^!]*$", ""));
181
+ }
182
+
183
+ File folder = new File(url.toURI()).getParentFile();
184
+ for (;; folder = folder.getParentFile()) {
185
+ if (folder == null) {
186
+ throw new RuntimeException("Cannot find 'embulk-filter-xxx' folder.");
187
+ }
188
+
189
+ if (folder.getName().startsWith("embulk-input-")) {
190
+ return folder;
191
+ }
192
+ }
193
+ } catch (URISyntaxException | MalformedURLException e) {
194
+ throw new RuntimeException(e);
195
+ }
196
+ }
197
+ }
data/src/main/java/org/embulk/filter/postgress_lookup/PostgressLookupFilterPlugin.java CHANGED
@@ -6,6 +6,8 @@ import org.embulk.config.*;
6
6
  import org.embulk.spi.*;
7
7
  import org.embulk.spi.time.Timestamp;
8
8
  import org.embulk.spi.type.Types;
9
+ import org.slf4j.Logger;
10
+ import org.slf4j.LoggerFactory;
9
11
 
10
12
  import java.sql.*;
11
13
  import java.time.Instant;
@@ -13,6 +15,8 @@ import java.util.*;
13
15
 
14
16
  public class PostgressLookupFilterPlugin
15
17
  implements FilterPlugin {
18
+ private static final Logger logger = LoggerFactory.getLogger(PostgressLookupFilterPlugin.class);
19
+
16
20
  public interface PluginTask
17
21
  extends Task {
18
22
 
@@ -25,7 +29,7 @@ public class PostgressLookupFilterPlugin
25
29
  @Config("database")
26
30
  public String getDatabase();
27
31
 
28
- @Config("table")
32
+ @Config("table_name")
29
33
  public String getTableName();
30
34
 
31
35
  @Config("username")
@@ -208,6 +212,9 @@ public class PostgressLookupFilterPlugin
208
212
  columnConfigList.add(columnConfig);
209
213
  }
210
214
 
215
+ Set<String> unmatchedData = new LinkedHashSet<>();
216
+ List<String> keyColumns = task.getMappingFrom();
217
+
211
218
  while (reader.nextRecord()) {
212
219
 
213
220
  int colNum = 0;
@@ -218,7 +225,7 @@ public class PostgressLookupFilterPlugin
218
225
 
219
226
  for (Column column : inputSchema.getColumns()) {
220
227
  if (reader.isNull(column)) {
221
- if (column.getName().equalsIgnoreCase(inputColumns.get(keyMap.get("Key")))) {
228
+ if ((keyMap.get("Key") < inputColumns.size()) && column.getName().equalsIgnoreCase(inputColumns.get(keyMap.get("Key")))) {
222
229
  searchingKeyData.add("");
223
230
  int key = keyMap.get("Key");
224
231
  keyMap.put("Key", ++key);
@@ -240,6 +247,8 @@ public class PostgressLookupFilterPlugin
240
247
  List<String> matchedData = new ArrayList<>();
241
248
  if (keyValuePair.containsKey(key)) {
242
249
  matchedData = keyValuePair.get(key);
250
+ }else{
251
+ unmatchedData.add(key);
243
252
  }
244
253
 
245
254
  if (matchedData.size() == 0) {
@@ -256,6 +265,19 @@ public class PostgressLookupFilterPlugin
256
265
  builder.addRecord();
257
266
  }
258
267
 
268
+ String info="\n--------------------Unmatched rows.....................\nMapping Key Columns: ";
269
+ for(int i=0;i<keyColumns.size();i++){
270
+ info+= keyColumns.get(i);
271
+ if(i!=keyColumns.size()-1){
272
+ info+=",";
273
+ }
274
+ }
275
+ info+="\n";
276
+ for(String key: unmatchedData){
277
+ info+= key+"\n";
278
+ }
279
+ logger.info(info);
280
+
259
281
  }
260
282
 
261
283
 
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: embulk-filter-postgres_lookup
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.5
4
+ version: 0.1.7
5
5
  platform: ruby
6
6
  authors:
7
7
  - Infoobjects Inc.
8
- autorequire:
8
+ autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2022-11-11 00:00:00.000000000 Z
11
+ date: 2023-03-03 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
@@ -49,14 +49,17 @@ files:
49
49
  - LICENSE
50
50
  - README.md
51
51
  - build.gradle
52
- - calendar.csv
53
- - classpath/embulk-filter-postgres_lookup-0.1.5.jar
52
+ - classpath/embulk-filter-postgres_lookup-0.1.7.jar
54
53
  - classpath/mysql-connector-java-8.0.19.jar
55
54
  - classpath/postgresql-9.0-801.jdbc4.jar
56
55
  - classpath/protobuf-java-3.6.1.jar
57
56
  - config/checkstyle/checkstyle.xml
58
57
  - config/checkstyle/default.xml
59
- - example/config.yml
58
+ - example/postgres-filter-lookup_testing.yml.liquid
59
+ - example/postgres-filter-lookup_testing_csv.yml.liquid
60
+ - example/postgres-filter-lookup_testing_mssql.yml.liquid
61
+ - example/postgres-filter-lookup_testing_mysql.yml.liquid
62
+ - example/postgres-filter-lookup_testing_oracle.yml.liquid
60
63
  - gradle/wrapper/gradle-wrapper.jar
61
64
  - gradle/wrapper/gradle-wrapper.properties
62
65
  - gradlew
@@ -66,11 +69,11 @@ files:
66
69
  - src/main/java/org/embulk/filter/postgress_lookup/PostGresConnection.java
67
70
  - src/main/java/org/embulk/filter/postgress_lookup/PostgressLookupFilterPlugin.java
68
71
  - src/test/java/org/embulk/filter/postgress_lookup/TestPostgressLookupFilterPlugin.java
69
- homepage:
72
+ homepage:
70
73
  licenses:
71
74
  - MIT
72
75
  metadata: {}
73
- post_install_message:
76
+ post_install_message:
74
77
  rdoc_options: []
75
78
  require_paths:
76
79
  - lib
@@ -85,8 +88,9 @@ required_rubygems_version: !ruby/object:Gem::Requirement
85
88
  - !ruby/object:Gem::Version
86
89
  version: '0'
87
90
  requirements: []
88
- rubygems_version: 3.2.33
89
- signing_key:
91
+ rubyforge_project:
92
+ rubygems_version: 2.7.6
93
+ signing_key:
90
94
  specification_version: 4
91
95
  summary: Postgress Lookup filter plugin for Embulk
92
96
  test_files: []
data/calendar.csv DELETED
@@ -1,10 +0,0 @@
1
- dim_calendar_key,year_number,quarter_number,attr_1
2
- -1,0,0
3
- 19900101,1990,1,FirstIndia
4
- 19900102,1990,2,SecondUSA
5
- 19900103,1990,,1000.56
6
- 19900104,1990,4,40_China
7
- 19900105,1990,5,50_Ukraine
8
- 19900106,1990,2,10_USA
9
- 19900107,1990,3,30_UK
10
- 19900109,1990,6,
data/example/config.yml DELETED
@@ -1,48 +0,0 @@
1
- exec:
2
- max_threads: 2
3
- min_output_tasks: 1
4
- in:
5
- type: file
6
- path_prefix: C:\Users\Abhishek Gupta\Desktop\github\embulk-filter-postgres_lookup\calendar.csv
7
- parser:
8
- type: csv
9
- columns:
10
- - { name: dim_calendar_key, type: long }
11
- - { name: year_number, type: long }
12
- - { name: quarter_number, type: long }
13
- - { name: attr_1, type: string }
14
- filters:
15
- - type: postgress_lookup
16
- driver_path: C:/Users/Abhishek Gupta/Desktop/postgresql-9.0-801.jdbc4.jar
17
- driver_class: org.postgresql.Driver
18
- schema_name: testSchema
19
- host: localhost
20
- port: 5432
21
- database: Abhishek
22
- table: country
23
- username: postgres
24
- password: root
25
- mapping_from:
26
- - quarter_number
27
- - attr_1
28
- mapping_to:
29
- - id
30
- - country_address
31
- new_columns:
32
- - { name: country_name, type: string }
33
- - { name: country_address, type: string }
34
- out:
35
- type: file
36
- path_prefix: C:\Users\Abhishek Gupta\Desktop\B\output.csv
37
- file_ext: csv
38
- formatter:
39
- type: csv
40
- delimiter: "\t"
41
- newline: CRLF
42
- newline_in_field: LF
43
- charset: UTF-8
44
- quote_policy: MINIMAL
45
- quote: '"'
46
- escape: "\\"
47
- null_string: "\\N"
48
- default_timezone: 'UTC'