embulk-output-snowflake 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 383ccdda739d086fd4ec83452382f75139b859b7
+  data.tar.gz: e115339e03ef287b89638c0411ffbafc24e4c2f9
+SHA512:
+  metadata.gz: da715c55401a0bf7b7009293196db2a115330e79ac935aa92ea3aa296d513c98d119886c2c743384c4bde23f823339543792bd16afcf3a4141aaa50e728c62df
+  data.tar.gz: 9de41d872c721e23de9e501fe89cbb6fea440cd8eaf0f9483b7ba31b1096e7ef4439a67a2ed9dcfa573c61dcf259df5929935567dcf82f6d0dcd6027c981b81b

data/README.md

@@ -0,0 +1,101 @@
+# Generic JDBC output plugin for Embulk
+
+Snowflake output plugin for Embulk loads records to a Snowflake using a JDBC driver.
+
+## Overview
+
+* **Plugin type**: output
+* **Load all or nothing**: depends on the mode. see below.
+* **Resume supported**: depends on the mode. see below.
+
+## Configuration
+
+- **driver_path**: path to the jar file of the Snowflake JDBC driver (e.g. 'snowflake-jdbc-3.8.0.jar') (string, optional)
+- **url**: URL of the JDBC connection (e.g. 'jdbc:snowflake://host.eu-central-1.snowflakecomputing.com/?db=development') (string, required)
+- **user**: database login user name (string, optional)
+- **password**: database login password (string, optional)
+- **schema**: destination schema name (string, default: use default schema)
+- **table**: destination table name (string, required)
+- **create_table_constraint**: table constraint added to `CREATE TABLE` statement, like `CREATE TABLE <table_name> (<column1> <type1>, <column2> <type2>, ..., <create_table_constraint>) <create_table_option>`.
+- **create_table_option**: table option added to `CREATE TABLE` statement, like `CREATE TABLE <table_name> (<column1> <type1>, <column2> <type2>, ..., <create_table_constraint>) <create_table_option>`.
+- **transaction_isolation**: transaction isolation level for each connection ("read_uncommitted", "read_committed", "repeatable_read" or "serializable"). if not specified, database default value will be used.
+- **options**: extra JDBC properties (hash, default: {})
+- **retry_limit**: max retry count for database operations (integer, default: 12). When intermediate table to create already created by another process, this plugin will retry with another table name to avoid collision.
+- **retry_wait**: initial retry wait time in milliseconds (integer, default: 1000 (1 second))
+- **max_retry_wait**: upper limit of retry wait, which will be doubled at every retry (integer, default: 1800000 (30 minutes))
+- **mode**: "insert", "insert_direct", "truncate_insert", or "replace". See below (string, required)
+- **batch_size**: size of a single batch insert (integer, default: 16777216)
+- **max_table_name_length**: maximum length of table name in this RDBMS (integer, default: 256)
+- **default_timezone**: If input column type (embulk type) is timestamp, this plugin needs to format the timestamp into a SQL string. This default_timezone option is used to control the timezone. You can overwrite timezone for each columns using column_options option. (string, default: `UTC`)
+- **column_options**: advanced: a key-value pairs where key is a column name and value is options for the column.
+  - **type**: type of a column when this plugin creates new tables (e.g. `VARCHAR(255)`, `INTEGER NOT NULL UNIQUE`). This used when this plugin creates intermediate tables (insert and truncate_insert modes), when it creates the target table (replace mode), and when it creates nonexistent target table automatically. (string, default: depends on input column type. `BIGINT` if input column type is long, `BOOLEAN` if boolean, `DOUBLE PRECISION` if double, `CLOB` if string, `TIMESTAMP` if timestamp)
+  - **value_type**: This plugin converts input column type (embulk type) into a database type to build a INSERT statement. This value_type option controls the type of the value in a INSERT statement. (string, default: depends on the sql type of the column. Available values options are: `byte`, `short`, `int`, `long`, `double`, `float`, `boolean`, `string`, `nstring`, `date`, `time`, `timestamp`, `decimal`, `json`, `null`, `pass`)
+  - **timestamp_format**: If input column type (embulk type) is timestamp and value_type is `string` or `nstring`, this plugin needs to format the timestamp value into a string. This timestamp_format option is used to control the format of the timestamp. (string, default: `%Y-%m-%d %H:%M:%S.%6N`)
+  - **timezone**: If input column type (embulk type) is timestamp, this plugin needs to format the timestamp value into a SQL string. In this cases, this timezone option is used to control the timezone. (string, value of default_timezone option is used by default)
+- **before_load**: if set, this SQL will be executed before loading all records. In truncate_insert mode, the SQL will be executed after truncating. replace mode doesn't support this option.
+- **after_load**: if set, this SQL will be executed after loading all records.
+
+## Modes
+
+* **insert**:
+  * Behavior: This mode writes rows to some intermediate tables first. If all those tasks run correctly, runs `INSERT INTO <target_table> SELECT * FROM <intermediate_table_1> UNION ALL SELECT * FROM <intermediate_table_2> UNION ALL ...` query. If the target table doesn't exist, it is created automatically.
+  * Transactional: Yes. This mode successfully writes all rows, or fails with writing zero rows.
+  * Resumable: Yes.
+* **insert_direct**:
+  * Behavior: This mode inserts rows to the target table directly. If the target table doesn't exist, it is created automatically.
+  * Transactional: No. If fails, the target table could have some rows inserted.
+  * Resumable: No.
+* **truncate_insert**:
+  * Behavior: Same with `insert` mode excepting that it truncates the target table right before the last `INSERT ...` query.
+  * Transactional: Yes.
+  * Resumable: Yes.
+* **replace**:
+  * Behavior: This mode writes rows to an intermediate table first. If all those tasks run correctly, drops the target table and alters the name of the intermediate table into the target table name.
+  * Transactional: No. If fails, the target table could be dropped.
+  * Resumable: No.
+* **merge**:
+  * Behavior: This mode writes rows to some intermediate tables first. If all those tasks run correctly, merges the intermediate tables into the target table. Namely, if primary keys of a record in the intermediate tables already exist in the target table, the target record is updated by the intermediate record, otherwise the intermediate record is inserted. If the target table doesn't exist, it is created automatically.
+  * Transactional: Yes.
+  * Resumable: Yes.
+* **merge_direct**:
+  * Behavior: This mode merges rows to the target table directly. Namely, if primary keys of an input record already exist in the target table, the target record is updated by the input record, otherwise the input record is inserted. If the target table doesn't exist, it is created automatically.
+  * Transactional: No.
+  * Resumable: No.
+
+## Example
+
+```yaml
+out:
+  type: snowflake
+  driver_path: /opt/snowflake-jdbc-3.8.0.jar
+  url: jdbc:snowflake://host.eu-central-1.snowflakecomputing.com/?db=development&warehouse=dwh&role=bi&schema=dwh
+  user: myuser
+  password: "mypassword"
+  table: my_table
+  mode: insert
+```
+
+Advanced configuration:
+
+```yaml
+out:
+  type: snowflake
+  driver_path: /opt/snowflake-jdbc-3.8.0.jar
+  url: jdbc:snowflake://host.eu-central-1.snowflakecomputing.com/?db=development&warehouse=dwh&role=bi&schema=dwh
+  user: myuser
+  password: "mypassword"
+  table: my_table
+  options: {loglevel: 2}
+  mode: insert_direct
+  column_options:
+    my_col_1: {type: 'VARCHAR(255)'}
+    my_col_3: {type: 'INT NOT NULL'}
+    my_col_4: {value_type: string, timestamp_format: `%Y-%m-%d %H:%M:%S %z`, timezone: '-0700'}
+    my_col_5: {type: 'DECIMAL(18,9)', value_type: pass}
+```
+
+## Build
+
+```
+$ ./gradlew gem
+```

data/build.gradle

@@ -0,0 +1,5 @@
+dependencies {
+    compile project(':embulk-output-jdbc')
+    
+    testCompile project(':embulk-output-jdbc').sourceSets.test.output
+}

data/lib/embulk/output/snowflake.rb

@@ -0,0 +1,3 @@
+Embulk::JavaPlugin.register_output(
+  :snowflake, "org.embulk.output.SnowflakeOutputPlugin",
+  File.expand_path('../../../../classpath', __FILE__))

data/src/main/java/org/embulk/output/SnowflakeOutputPlugin.java

@@ -0,0 +1,165 @@
+package org.embulk.output;
+
+import java.sql.*;
+import java.util.Properties;
+import java.util.List;
+import java.io.IOException;
+import java.util.Locale;
+
+import com.google.common.base.Optional;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.ImmutableList;
+
+
+import org.embulk.config.Config;
+import org.embulk.config.ConfigDefault;
+import org.embulk.output.jdbc.*;
+import org.embulk.output.snowflake.SnowflakeOutputConnector;
+import org.embulk.spi.Schema;
+import org.embulk.output.snowflake.SnowflakeOutputConnection;
+
+public class SnowflakeOutputPlugin
+        extends AbstractJdbcOutputPlugin
+{
+    public interface GenericPluginTask extends PluginTask
+    {
+        @Config("driver_path")
+        @ConfigDefault("null")
+        public Optional<String> getDriverPath();
+
+        @Config("url")
+        public String getUrl();
+
+        @Config("user")
+        @ConfigDefault("null")
+        public Optional<String> getUser();
+
+        @Config("password")
+        @ConfigDefault("null")
+        public Optional<String> getPassword();
+
+        @Config("schema")
+        @ConfigDefault("null")
+        public Optional<String> getSchema();
+
+        @Config("max_table_name_length")
+        @ConfigDefault("30")
+        public int getMaxTableNameLength();
+    }
+
+    @Override
+    protected Class<? extends PluginTask> getTaskClass()
+    {
+        return GenericPluginTask.class;
+    }
+
+    @Override
+    protected Features getFeatures(PluginTask task)
+    {
+        GenericPluginTask t = (GenericPluginTask) task;
+        return new Features()
+            .setMaxTableNameLength(t.getMaxTableNameLength())
+            .setSupportedModes(ImmutableSet.of(Mode.INSERT, Mode.INSERT_DIRECT, Mode.TRUNCATE_INSERT, Mode.REPLACE));
+    }
+
+    @Override
+    protected SnowflakeOutputConnector getConnector(PluginTask task, boolean retryableMetadataOperation)
+    {
+        GenericPluginTask t = (GenericPluginTask) task;
+
+        if (t.getDriverPath().isPresent()) {
+            addDriverJarToClasspath(t.getDriverPath().get());
+        }
+
+        Properties props = new Properties();
+
+        props.putAll(t.getOptions());
+
+        if (t.getUser().isPresent()) {
+            props.setProperty("user", t.getUser().get());
+        }
+        if (t.getPassword().isPresent()) {
+            props.setProperty("password", t.getPassword().get());
+        }
+        logConnectionProperties(t.getUrl(), props);
+
+        return new SnowflakeOutputConnector(t.getUrl(), props,
+                t.getSchema().orNull(), t.getTransactionIsolation());
+    }
+
+    @Override
+    protected BatchInsert newBatchInsert(PluginTask task, Optional<MergeConfig> mergeConfig) throws IOException, SQLException
+    {
+        return new StandardBatchInsert(getConnector(task, true), mergeConfig);
+    }
+
+    @Override
+    protected void doBegin(JdbcOutputConnection con,
+                           PluginTask task, final Schema schema, int taskCount) throws SQLException
+    {
+
+        SnowflakeOutputConnection snowflakeCon = (SnowflakeOutputConnection)con;
+
+        super.doBegin(snowflakeCon, task, schema, taskCount);
+    }
+
+    @Override
+    public Optional<JdbcSchema> newJdbcSchemaFromTableIfExists(JdbcOutputConnection connection,
+                                                               TableIdentifier table) throws SQLException
+    {
+        if (!connection.tableExists(table)) {
+            // DatabaseMetaData.getPrimaryKeys fails if table does not exist
+            return Optional.absent();
+        }
+
+        DatabaseMetaData dbm = connection.getMetaData();
+        String escape = dbm.getSearchStringEscape();
+        String catalog = dbm.getConnection().getCatalog();
+
+        ResultSet rs = dbm.getPrimaryKeys(catalog, table.getSchemaName(), table.getTableName());
+        ImmutableSet.Builder<String> primaryKeysBuilder = ImmutableSet.builder();
+        try {
+            while(rs.next()) {
+                primaryKeysBuilder.add(rs.getString("COLUMN_NAME"));
+            }
+        } finally {
+            rs.close();
+        }
+        ImmutableSet<String> primaryKeys = primaryKeysBuilder.build();
+
+        ImmutableList.Builder<JdbcColumn> builder = ImmutableList.builder();
+        rs = dbm.getColumns(
+                catalog,
+                JdbcUtils.escapeSearchString(table.getSchemaName(), escape),
+                JdbcUtils.escapeSearchString(table.getTableName(), escape),
+                null);
+        try {
+            while (rs.next()) {
+                String columnName = rs.getString("COLUMN_NAME");
+                String simpleTypeName = rs.getString("TYPE_NAME").toUpperCase(Locale.ENGLISH);
+                boolean isUniqueKey = primaryKeys.contains(columnName);
+                int sqlType = rs.getInt("DATA_TYPE");
+                int colSize = rs.getInt("COLUMN_SIZE");
+                int decDigit = rs.getInt("DECIMAL_DIGITS");
+                if (rs.wasNull()) {
+                    decDigit = -1;
+                }
+                int charOctetLength = rs.getInt("CHAR_OCTET_LENGTH");
+                boolean isNotNull = "NO".equals(rs.getString("IS_NULLABLE"));
+                //rs.getString("COLUMN_DEF") // or null  // TODO
+                builder.add(JdbcColumn.newGenericTypeColumn(
+                        columnName, sqlType, simpleTypeName, colSize, decDigit, charOctetLength, isNotNull, isUniqueKey));
+                // We can't get declared column name using JDBC API.
+                // Subclasses need to overwrite it.
+            }
+        } finally {
+            rs.close();
+        }
+        List<JdbcColumn> columns = builder.build();
+        if (columns.isEmpty()) {
+            return Optional.absent();
+        } else {
+            return Optional.of(new JdbcSchema(columns));
+        }
+    }
+}

data/src/main/java/org/embulk/output/snowflake/SnowflakeOutputConnection.java

@@ -0,0 +1,51 @@
+package org.embulk.output.snowflake;
+
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+
+import org.embulk.output.jdbc.JdbcOutputConnection;
+import org.embulk.output.jdbc.JdbcUtils;
+import org.embulk.output.jdbc.TableIdentifier;
+
+public class SnowflakeOutputConnection
+        extends JdbcOutputConnection
+{
+    public SnowflakeOutputConnection(Connection connection, String schemaName)
+        throws SQLException
+    {
+        super(connection, schemaName);
+    }
+
+    @Override
+    public boolean tableExists(TableIdentifier table) throws SQLException
+    {
+        String schemaName = JdbcUtils.escapeSearchString(table.getSchemaName(), connection.getMetaData().getSearchStringEscape());
+        String database = connection.getCatalog();
+        try (ResultSet rs = connection.getMetaData().getTables(database, schemaName, table.getTableName(), null)) {
+            return rs.next();
+        }
+    }
+
+    @Override
+    public boolean tableExists(String tableName) throws SQLException
+    {
+        return tableExists(new TableIdentifier(connection.getCatalog(), schemaName, tableName));
+    }
+
+    @Override
+    protected void setSearchPath(String schema) throws SQLException
+    {
+        Statement stmt = connection.createStatement();
+        try {
+            String sql = "USE SCHEMA " + quoteIdentifierString(schema);
+            executeUpdate(stmt, sql);
+            commitIfNecessary(connection);
+        } finally {
+            stmt.close();
+        }
+    }
+
+
+}

data/src/main/java/org/embulk/output/snowflake/SnowflakeOutputConnector.java

@@ -0,0 +1,48 @@
+package org.embulk.output.snowflake;
+
+import com.google.common.base.Optional;
+import org.embulk.output.jdbc.AbstractJdbcOutputConnector;
+import org.embulk.output.jdbc.JdbcOutputConnection;
+import org.embulk.output.jdbc.TransactionIsolation;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.Properties;
+
+public class SnowflakeOutputConnector
+        extends AbstractJdbcOutputConnector
+{
+    private final String url;
+    private final Properties properties;
+    private final String schemaName;
+
+    public SnowflakeOutputConnector(String url, Properties properties, String schemaName,
+                                    Optional<TransactionIsolation> transactionIsolation)
+    {
+        super(transactionIsolation);
+        try {
+            Class.forName("com.snowflake.client.jdbc.SnowflakeDriver");
+        } catch (Exception ex) {
+            throw new RuntimeException(ex);
+        }
+        this.url = url;
+        this.properties = properties;
+        this.schemaName = schemaName;
+    }
+
+    @Override
+    protected JdbcOutputConnection connect() throws SQLException
+    {
+        Connection c = DriverManager.getConnection(url, properties);
+        try {
+            SnowflakeOutputConnection con = new SnowflakeOutputConnection(c, schemaName);
+            c = null;
+            return con;
+        } finally {
+            if (c != null) {
+                c.close();
+            }
+        }
+    }
+}

metadata

@@ -0,0 +1,52 @@
+--- !ruby/object:Gem::Specification
+name: embulk-output-snowflake
+version: !ruby/object:Gem::Version
+  version: 0.1.1
+platform: ruby
+authors:
+- Piotr Wilkowski
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2019-05-29 00:00:00.000000000 Z
+dependencies: []
+description: Inserts or updates records to a table.
+email:
+- piwil@wp.pl
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- README.md
+- build.gradle
+- classpath/embulk-output-jdbc-0.1.1.jar
+- classpath/embulk-output-snowflake-0.1.1.jar
+- lib/embulk/output/snowflake.rb
+- src/main/java/org/embulk/output/SnowflakeOutputPlugin.java
+- src/main/java/org/embulk/output/snowflake/SnowflakeOutputConnection.java
+- src/main/java/org/embulk/output/snowflake/SnowflakeOutputConnector.java
+homepage: ''
+licenses:
+- Apache 2.0
+metadata: {}
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 2.4.8
+signing_key:
+specification_version: 4
+summary: Snowflake output plugin for Embulk
+test_files: []