redshift_connector 8.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 6dfd6aa65bfc44fca5d3e0dc7e042911344be4213764e267136f7259b3529a90
+  data.tar.gz: f12016b3044c1199e0b0e3bba4c8e8826e6a72b80d40088d849110dd4624607f
+SHA512:
+  metadata.gz: 68e8a3265255b168cbb40b1b200902922fb9641035533cf561ab69feee1d825d15dd999f3d0fb9dfbfc0b00349be662d713c1ca5ac3da418d49c44ad03771e20
+  data.tar.gz: 59cf47cc0aaec30d8025497dc11347904d50f9be8a5e68086d9c2952f773619b27be3c8ed5edf8a9f6708afe370ba1dfac7cb48c1602c7cbbb23d33e0457b054

data/.gitignore

@@ -0,0 +1,20 @@
+# Packaging
+*.gem
+/spec/reports/
+/spec/examples.txt
+
+# Documents
+/.yardoc/
+/_yardoc/
+/doc/
+/rdoc/
+
+# Bundler
+/.bundle/
+/vendor/bundle
+/lib/bundler/man/
+Gemfile.lock
+
+# Gem Specific
+test/database.yml
+test/config.rb

data/Gemfile

@@ -0,0 +1,7 @@
+source "https://rubygems.org"
+gemspec
+
+# We need explicit version specification (4 or 5) here,
+# to resolve version dependencies correctly.
+gem 'activerecord', '~> 5.0'
+gem 'mysql2'

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2016,2017 Minero Aoki
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,42 @@
+# Redshift Connector for Rails
+
+redshift_connector is a Redshift bulk data connector for Rails (ActiveRecord).
+
+This library is formerly called as "redshift-connector",
+but renamed to "redshift_connector" to follow the gem naming standard.
+
+## Settings
+
+Add following block to your Gemfile and bundle.
+```
+gem 'redshift_connector'
+```
+Add config/initializers/redshift_connector.rb like following:
+```
+module RedshiftConnector
+  Exporter.default_data_source = Any_ActiveRecord_Class_Bound_To_Redshift
+
+  S3Bucket.add('primary', default: true,
+    region: 'YOUR_AWS_REGION_NAME',
+    bucket: 'YOUR_BUCKET_NAME',
+    prefix: 'YOUR_PREFIX',
+    iam_role: 'arn:aws:iam::XXXXXXXXXXXX:role/RedshiftReadOnly'
+    # For explicit S3 access, use following:
+    # aws_access_key_id: 'XXXXXXXXXXXXX',
+    # aws_secret_access_key: 'XXXXXXXXXXXXX'
+  )
+end
+```
+
+## Usage
+
+### Fetching rows
+
+```
+RedshiftConnector.foreach(schema: 'app_mst', table: 'shops', query: 'select id, name from app_mst.shops') do |id, name|
+  p [id, name]
+end
+```
+`schema` and `table` is the source table name (written in the query).
+This method executes Redshift UNLOAD statement with given query and
+unload result to the intermediate S3, then read contents.

data/RELEASE.md

@@ -0,0 +1,89 @@
+# Release Note
+
+## version 8.0.0
+
+- [INCOMPATIBLE] This library is renamed to "redshift_connector".  Just modify your Gemfile from "redshift-connector" to "redshift_connector".
+- [INCOMPATIBLE] redshift-connector-data_file gem is merged.
+- [INCOMPATIBLE] (internal) *DataFileBundle#each, #each_row, #each_object, #each_batch, #all_data_objects are removed.  Use DataFileBundleReader class instead.
+- [INCOMPATIBLE] (internal) AbstractDataFileBundle class is removed.
+- [INCOMPATIBLE] (internal) AbstractDataFile class is removed.
+
+## version 7.2.2
+
+- [fix] RedshiftConnector.transport_all: src_table/dest_table parameter did not work.
+- [fix] RedshiftConnector.transport_all (strategy=rename): newer activerecord-import requires class name.
+
+## version 7.2.1
+
+- no change.
+
+## version 7.2.0
+
+- Removes aws-sdk dependency
+
+## version 7.0.2
+
+- [fix] RedshiftConnector.foreach did not work
+
+## version 7.0.1
+
+- [fix] RedshiftConnector.transport_delta_from_s3, .transport_all_from_s3 were wrongly dropped, restore them.
+
+## version 7.0.0
+
+- [INCOMPATIBLE] Library hierarchy changed: redshift-connector/* -> redshift_connector/*.  redshift-connector.rb still exists as an entry point for bundler.
+- [new] Exporter becomes pluggable.  You can implement your own exporter data source instead of ActiveRecord.
+
+## version 6.0.0
+
+- version number change only.
+
+## version 5.6.0
+
+- Unifies version 4.x (supports Rails 4) and 5.x (supports Rails 5).
+
+## version 4.5.0 / 5.5.0
+
+- [new] Separates S3 access layer to another gem: redshift-connector-data_file
+
+## version 4.4.1 / 5.4.1
+
+- [new] New option enable_sort for Connector.foreach, to enforce global sorting.
+
+## version 4.4.0 / 5.4.0
+
+- [CHANGE] Drops export-only-once feature (and FORCE environment switch), it is not so useful.
+  Exporter now always exports data.
+
+## version 4.3.2 / 5.3.2
+
+- [new] Allows reading from S3 signed URL (for separated export/import processes)
+
+## version 4.3.1 / 5.3.1
+
+- First release for Rails 5 series.
+- [fix] Add option for AWS multi-regions support
+
+## version 4.3.0
+
+- [new] New method RedshiftConnector.foreach to read rows with UNLOAD
+
+## version 4.2.0
+
+- [new] New methods RedshiftConnector.transport_delta_from_s3, .transport_all_from_s3 to read from S3
+ 
+## version 4.1.0
+
+- [new] Introduces rebuild operator.  New facade method Connector.transport_all.
+  
+## version 4.0.2
+  
+- [fix] Correctly parses UNLOAD-generated CSV (dangerous characeters are escaped by backslash).
+  
+## version 4.0.1
+  
+- [new] Allow configure the default logger by RedshiftConnector.logger=.
+  
+## version 4.0.0
+  
+First release for Rails 4 series.

data/Rakefile

@@ -0,0 +1,3 @@
+task :test do
+  load "#{__dir__}/test/all.rb"
+end

data/lib/redshift_connector.rb

@@ -0,0 +1,35 @@
+module RedshiftConnector
+end
+
+require 'redshift_connector/connector'
+require 'redshift_connector/exporter'
+require 'redshift_connector/active_record_data_source'
+require 'redshift_connector/active_record_exporter'
+require 'redshift_connector/immediate_exporter'
+require 'redshift_connector/importer'
+require 'redshift_connector/s3_bucket'
+require 'redshift_connector/s3_data_file_bundle'
+require 'redshift_connector/exception'
+require 'redshift_connector/version'
+
+module RedshiftConnector
+  def RedshiftConnector.transport_delta(**params)
+    Connector.transport_delta(**params)
+  end
+
+  def RedshiftConnector.transport_all(**params)
+    Connector.transport_all(**params)
+  end
+
+  def RedshiftConnector.transport_delta_from_s3(**params)
+    Connector.transport_delta_from_s3(**params)
+  end
+
+  def RedshiftConnector.transport_all_from_s3(**params)
+    Connector.transport_all_from_s3(**params)
+  end
+
+  def RedshiftConnector.foreach(**params, &block)
+    Exporter.foreach(**params, &block)
+  end
+end

data/lib/redshift_connector/active_record_data_source.rb

@@ -0,0 +1,23 @@
+require 'redshift_connector/exporter_builder'
+
+module RedshiftConnector
+  class ActiveRecordDataSource
+    def ActiveRecordDataSource.for_dao(dao)
+      new(dao)
+    end
+
+    def initialize(dao)
+      @dao = dao
+    end
+
+    def exporter_builder
+      ExporterBuilder.new(ds: self, exporter_class: ActiveRecordExporter)
+    end
+
+    def execute_query(query_str)
+      @dao.connection_pool.with_connection {|conn|
+        conn.execute(query_str)
+      }
+    end
+  end
+end

data/lib/redshift_connector/active_record_exporter.rb

@@ -0,0 +1,47 @@
+require 'redshift_connector/s3_data_file_bundle'
+require 'redshift_connector/query'
+require 'redshift_connector/logger'
+
+module RedshiftConnector
+  class ActiveRecordExporter
+    def initialize(ds:, query:, bundle_params:, enable_sort: false, logger: RedshiftConnector.logger)
+      @ds = ds
+      @query = query
+      @bundle_params = bundle_params
+      @enable_sort = enable_sort
+      @logger = logger
+
+      @bundle = S3DataFileBundle.for_params(bundle_params)
+    end
+
+    attr_reader :query
+    attr_reader :bundle_params
+    attr_reader :bundle
+    attr_reader :logger
+
+    def execute
+      @bundle.clear
+      unload_query = UnloadQuery.new(query: @query, bundle: @bundle, enable_sort: @enable_sort)
+      @logger.info "EXPORT #{unload_query.description} -> #{@bundle.url}*"
+      stmt = unload_query.to_sql
+      @logger.info "[SQL/Redshift] #{batch_job_label}#{stmt.strip}"
+      @ds.execute_query(batch_job_label + stmt)
+      @bundle
+    end
+
+    def batch_job_label
+      @batch_job_label ||= begin
+        components = Dir.getwd.split('/')
+        app = if components.last == 'current'
+            # is Capistrano environment
+            components[-2]
+          else
+            components[-1]
+          end
+        batch_file = caller.detect {|c| /redshift_connector|active_record/ !~ c }
+        path = batch_file ? batch_file.split(':').first : '?'
+        "/* Job: #{app}:#{path} */ "
+      end
+    end
+  end
+end

data/lib/redshift_connector/connector.rb

@@ -0,0 +1,189 @@
+require 'redshift_connector/exporter'
+require 'redshift_connector/immediate_exporter'
+require 'redshift_connector/importer'
+require 'redshift_connector/s3_data_file_bundle'
+require 'redshift_connector/data_file_bundle_params'
+require 'redshift_connector/data_file_bundle_reader'
+require 'redshift_connector/logger'
+
+module RedshiftConnector
+  class Connector
+    def Connector.transport_delta_from_s3(
+      bucket: nil,
+      prefix:,
+      format:,
+      filter: nil,
+      table:,
+      columns:,
+      delete_cond: nil,
+      upsert_columns: nil,
+      logger: RedshiftConnector.logger,
+      quiet: false
+    )
+      logger = NullLogger.new if quiet
+      bundle = S3DataFileBundle.for_prefix(
+        bucket: (bucket ? S3Bucket.get(bucket) : S3Bucket.default),
+        prefix: prefix,
+        format: format,
+        logger: logger
+      )
+      exporter = ImmediateExporter.new(bundle: bundle, logger: logger)
+      importer = Importer.for_delta_upsert(
+        table: table,
+        columns: columns,
+        delete_cond: delete_cond,
+        upsert_columns: upsert_columns,
+        logger: logger
+      )
+      new(exporter: exporter, importer: importer, filter: filter, logger: logger)
+    end
+
+    def Connector.transport_delta(
+        schema:,
+        table: nil,
+        src_table: table,
+        dest_table: table,
+        condition:,
+        columns:,
+        delete_cond: nil,
+        upsert_columns: nil,
+        bucket: nil,
+        txn_id: nil,
+        filter:,
+        logger: RedshiftConnector.logger,
+        quiet: false
+    )
+      unless src_table and dest_table
+        raise ArgumentError, "missing :table, :src_table or :dest_table"
+      end
+      logger = NullLogger.new if quiet
+      bundle_params = DataFileBundleParams.new(
+        bucket: bucket,
+        schema: schema,
+        table: src_table,
+        txn_id: txn_id,
+        logger: logger
+      )
+      exporter = Exporter.for_table_delta(
+        bundle_params: bundle_params,
+        schema: schema,
+        table: src_table,
+        columns: columns,
+        condition: condition,
+        logger: logger
+      )
+      importer = Importer.for_delta_upsert(
+        table: dest_table,
+        columns: columns,
+        delete_cond: delete_cond,
+        upsert_columns: upsert_columns,
+        logger: logger
+      )
+      new(exporter: exporter, importer: importer, filter: filter, logger: logger)
+    end
+
+    def Connector.transport_all_from_s3(
+        strategy: 'rename',
+        table:,
+        columns:,
+        bucket: nil,
+        prefix:,
+        format:,
+        filter: nil,
+        logger: RedshiftConnector.logger,
+        quiet: false
+    )
+      logger = NullLogger.new if quiet
+      bundle = S3DataFileBundle.for_prefix(
+        bucket: (bucket ? S3Bucket.get(bucket) : S3Bucket.default),
+        prefix: prefix,
+        format: format,
+        logger: logger
+      )
+      exporter = ImmediateExporter.new(bundle: bundle, logger: logger)
+      importer = Importer.for_rebuild(
+        strategy: strategy,
+        table: table,
+        columns: columns,
+        logger: logger
+      )
+      new(exporter: exporter, importer: importer, filter: filter, logger: logger)
+    end
+
+    def Connector.transport_all(
+        strategy: 'rename',
+        schema:,
+        table: nil,
+        src_table: table,
+        dest_table: table,
+        columns:,
+        bucket: nil,
+        txn_id: nil,
+        filter:,
+        logger: RedshiftConnector.logger,
+        quiet: false
+    )
+      logger = NullLogger.new if quiet
+      bundle_params = DataFileBundleParams.new(
+        bucket: bucket,
+        schema: schema,
+        table: src_table,
+        txn_id: txn_id,
+        logger: logger
+      )
+      exporter = Exporter.for_table(
+        bundle_params: bundle_params,
+        schema: schema,
+        table: src_table,
+        columns: columns,
+        logger: logger
+      )
+      importer = Importer.for_rebuild(
+        strategy: strategy,
+        table: dest_table,
+        columns: columns,
+        logger: logger
+      )
+      new(exporter: exporter, importer: importer, filter: filter, logger: logger)
+    end
+
+    def initialize(exporter:, importer:, filter: nil, logger:)
+      @exporter = exporter
+      @importer = importer
+      @filter = filter
+      @logger = logger
+      @bundle = nil
+    end
+
+    def export_enabled?
+      not ENV['IMPORT_ONLY']
+    end
+
+    def import_enabled?
+      not ENV['EXPORT_ONLY']
+    end
+
+    def execute
+      export if export_enabled?
+      import if import_enabled?
+    end
+
+    def export
+      @logger.info "==== export task =================================================="
+      @bundle = @exporter.execute
+    end
+
+    DEFAULT_BATCH_SIZE = 1000
+
+    def import
+      @logger.info "==== import task =================================================="
+      r = DataFileBundleReader.new(
+        @bundle,
+        filter: @filter,
+        batch_size: DEFAULT_BATCH_SIZE,
+        logger: @logger
+      )
+      @importer.execute(r)
+    end
+  end
+end