bigshift 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 6b933f1227d7a30c817577db6ca2f1517111d0e2
+  data.tar.gz: c53b1f16c4977e04c796a5f645d3d4ca600e3b13
+SHA512:
+  metadata.gz: dc549cf4e6ec70de381ff11118967f68c3d6868aa3892656d379d265d3669f787a81b1193b1e605c0f84f8b692e75a51c5bf45d15e68bc7b43843047c22650e0
+  data.tar.gz: 3c5407a160e9389e478c2b9c2c4f8561ffdcb64d038514bf5e6b41c4dc78dc83ba412c6aa31b1a9a9a24217525eb43b3ff54622cdef95053949031a0fbf11096

data/LICENSE.txt

@@ -0,0 +1,12 @@
+Copyright (c) 2014, Burt AB
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

data/README.md

@@ -0,0 +1,101 @@
+# BigShift
+
+BigShift is a tool for moving tables from Redshift to BigQuery. It will create a table in BigQuery with a schema that matches the Redshift table, dump the data to S3, transfer it to GCS and finally load it into the BigQuery table.
+
+# Installation
+
+```
+$ gem install bigshift
+```
+
+# Requirements
+
+On the AWS side you need a Redshift cluster and an S3 bucket, and credentials that let you read from Redshift, and read and write to the S3 bucket (it doesn't have to be to the whole bucket, a prefix works fine). On the GCP side you need a Cloud Storage bucket, a BigQuery dataset and credentials that allows reading and writing to the bucket, and create BigQuery tables.
+
+# Usage
+
+The main interface to BigShift is the `bigshift` command line tool.
+
+BigShift can also be used as a library in a Ruby application. Look at the tests, and how the `bigshift` tool is built to figure out how.
+
+## Arguments
+
+Running `bigshift` without any arguments, or with `--help` will show the options. All except `--s3-prefix` are required.
+
+### GCP credentials
+
+The `--gcp-credentials` argument must be a path to a JSON file that contains a public/private key pair for a GCP user. The best way to obtain this is to create a new service account and chose JSON as the key type when prompted.
+
+### AWS credentials
+
+The `--aws-credentials` argument must be a path to a JSON or YAML file that contains `aws_access_key_id` and `aws_secret_access_key`, and optionally `token`.
+
+```yaml
+---
+aws_access_key_id: AKXYZABC123FOOBARBAZ
+aws_secret_access_key: eW91ZmlndXJlZG91dGl0d2FzYmFzZTY0ISEhCg
+```
+
+These credentials need to be allowed to read and write the S3 location you specify with `--s3-bucket` and `--s3-prefix`.
+
+### Redshift credentials
+
+The `--rs-credentials` argument must be a path to a JSON or YAML file that contains the `host` and `port` of the Redshift cluster, as well as the `username` and `password` required to connect.
+
+```yaml
+---
+host: my-cluster.abc123.eu-west-1.redshift.amazonaws.com
+port: 5439
+username: my_redshift_user
+password: dGhpc2lzYWxzb2Jhc2U2NAo
+```
+
+# How does it work?
+
+There are four main pieces to BigShift: the Redshift unloader, the transfer, the BigQuery load and the schema translation.
+
+In theory it's pretty simple: the Redshift table is dumped to S3 using Redshift's `UNLOAD` command, copied over to GCS and loaded into BigQuery – but the devil is the details.
+
+The CSV produced by Redshift's `UNLOAD` can't be loaded into BigQuery no matter what options you specify on either end. Redshift can quote _all_ fields or none, but BigQuery doesn't allow non-string fields to be quoted. The format of booleans and timestamps are not compatible, and they expect quotes in quoted fields to be escaped differently, to name a few things.
+
+This means that a lot of what BigShift does is make sure that the data that is dumped from Redshift is compatible with BigQuery. To do this it reads the table schema and translates the different datatypes while the data is dumped. Quotes are escaped, timestamps formatted, and so on.
+
+Once the data is on S3 it's fairly simple to move it over to GCS. GCS has a great service called Transfer Service, that does the transfer for you. If this didn't exist you would have to stream all of the bytes through the machine that ran BigShift. As long as you've set up the credentials right in AWS IAM this works smoothly.
+
+Once the data is in GCS, the BigQuery table can be created and loaded. At this point the Redshift table's schema is translated into a BigQuery schema. The Redshift datatypes are mapped to BigQuery datatypes and things like nullability are determines. The mapping is straighforward:
+
+* `BOOLEAN` in Redshift becomes `BOOLEAN` in BigQuery
+* all Redshift integer types are mapped to BigQuery's `INTEGER`
+* all Redshift floating point types are mapped to BigQuery's `FLOAT`
+* `DATE` in Redshift becomes `STRING` in BigQuery (formatted as YYYY-MM-DD)
+* `NUMERIC` is mapped to `STRING`, because BigQuery doesn't have any equivalent data type and using `STRING` avoids loosing precision
+* `TIMESTAMP` in Redshift becomes `TIMESTAMP` in BigQuery, and the data is transferred as a UNIX timestamp with fractional seconds (to the limit of what Redshift's `TIMESTAMP` datatype provides)
+* `CHAR` and `VARCHAR` obviously become `STRING` in BigQuery
+
+`NOT NULL` becomes `REQUIRED` in BigQuery, and `NULL` becomes `NULLABLE`.
+
+# What doesn't it do?
+
+* Currently BigShift doesn't delete the dumped table from S3 or from GCS. This is planned.
+* BigShift can't currently append to an existing BigQuery table. This feature would be possible to add.
+* The tool will happily overwrite any data on S3, GCS and in BigQuery that happen to be in the way (i.e. in the specified S3 or GCS location, or in the target table). This is convenient if you want to move the same data multiple times, but very scary and unsafe. To clobber everything will be an option in the future, but the default will be much safer.
+* There is no transformation or processing of the data. When moving to BigQuery you might want to split a string and use the pieces as values in a repeated field, but BigShift doesn't help you with that. You will almost always have to do some post processing in BigQuery once the data has been moved. Processing on the way would require a lot more complexity and involve either Hadoop or Dataflow, and that's beyond the scope of a tool like this.
+* BigShift can't move data back from BigQuery to Redshift. It can probably be done, but you would probably have to write a big part of the Redshift schema yourself since BigQuery's data model is so much simpler. Going from Redshift to BigQuery is simple, most of Redshifts datatypes map directly to one of BigQuery's, and there's no encodings, sort or dist keys to worry about. Going in the other direction the tool can't know whether or not a `STRING` column in BigQuery should be a `CHAR(12)` or `VARCHAR(65535)`, and if it should be encoded as `LZO` or `BYTEDICT` or what should be the primary, sort, and dist key of the table.
+
+# Troubleshooting
+
+### I get SSL errors
+
+The certificates used by the Google APIs might not be installed on your system, try this as a workaround:
+
+```
+export SSL_CERT_FILE="$(find $GEM_HOME/gems -name 'google-api-client-*' | tail -n 1)/lib/cacerts.pem"
+```
+
+### I get errors when the data is loaded into BigQuery
+
+This could be anything, but it could be things that aren't escaped properly when the data is dumped from Redshift. Try figuring out from the errors where the problem is and what the data looks like and open an issue. The more you can figure out yourself the more likely it is that you will get help. No one wants to trawl through your data, make an effort.
+
+# Copyright
+
+© 2016 Theo Hultberg and contributors, see LICENSE.txt (BSD 3-Clause).

data/bin/bigshift

@@ -0,0 +1,18 @@
+#!/usr/bin/env ruby
+
+require 'bigshift/cli'
+
+begin
+  BigShift::Cli.new(ARGV).run
+rescue BigShift::CliError => e
+  $stderr.puts("#{e.message}:")
+  $stderr.puts
+  e.details.each do |detail|
+    $stderr.puts("* #{detail}")
+  end
+  $stderr.puts
+  $stderr.puts(e.usage)
+  $stderr.puts
+  exit(1)
+end
+

data/lib/bigshift.rb

@@ -0,0 +1,29 @@
+require 'google/apis/bigquery_v2'
+require 'google/apis/storagetransfer_v1'
+
+module BigShift
+  BigShiftError = Class.new(StandardError)
+
+  class NullLogger
+    def close(*); end
+    def debug(*); end
+    def debug?; false end
+    def error(*); end
+    def error?; false end
+    def fatal(*); end
+    def fatal?; false end
+    def info(*); end
+    def info?; false end
+    def unknown(*); end
+    def warn(*); end
+    def warn?; false end
+
+    INSTANCE = new
+  end
+end
+
+require 'bigshift/big_query/dataset'
+require 'bigshift/big_query/table'
+require 'bigshift/redshift_table_schema'
+require 'bigshift/redshift_unloader'
+require 'bigshift/cloud_storage_transfer'

data/lib/bigshift/big_query/dataset.rb

@@ -0,0 +1,41 @@
+module BigShift
+  module BigQuery
+    class Dataset
+      def initialize(big_query_service, project_id, dataset_id, options={})
+        @big_query_service = big_query_service
+        @project_id = project_id
+        @dataset_id = dataset_id
+        @logger = options[:logger] || NullLogger::INSTANCE
+      end
+
+      def table(table_name)
+        table_data = @big_query_service.get_table(@project_id, @dataset_id, table_name)
+        Table.new(@big_query_service, table_data, logger: @logger)
+      rescue Google::Apis::ClientError => e
+        if e.status_code == 404
+          nil
+        else
+          raise
+        end
+      end
+
+      def create_table(table_name, options={})
+        table_reference = Google::Apis::BigqueryV2::TableReference.new(
+          project_id: @project_id,
+          dataset_id: @dataset_id,
+          table_id: table_name
+        )
+        if options[:schema]
+          fields = options[:schema]['fields'].map { |f| Google::Apis::BigqueryV2::TableFieldSchema.new(name: f['name'], type: f['type'], mode: f['mode']) }
+          schema = Google::Apis::BigqueryV2::TableSchema.new(fields: fields)
+        end
+        table_spec = {}
+        table_spec[:table_reference] = table_reference
+        table_spec[:schema] = schema if schema
+        table_data = Google::Apis::BigqueryV2::Table.new(table_spec)
+        table_data = @big_query_service.insert_table(@project_id, @dataset_id, table_data)
+        Table.new(@big_query_service, table_data, logger: @logger)
+      end
+    end
+  end
+end

data/lib/bigshift/big_query/table.rb

@@ -0,0 +1,74 @@
+module BigShift
+  module BigQuery
+    class Table
+      def initialize(big_query_service, table_data, options={})
+        @big_query_service = big_query_service
+        @table_data = table_data
+        @logger = options[:logger] || NullLogger::INSTANCE
+        @thread = options[:thread] || Kernel
+      end
+
+      def load(uri, options={})
+        poll_interval = options[:poll_interval] || DEFAULT_POLL_INTERVAL
+        load_configuration = {}
+        load_configuration[:source_uris] = [uri]
+        load_configuration[:write_disposition] = options[:allow_overwrite] ? 'WRITE_TRUNCATE' : 'WRITE_EMPTY'
+        load_configuration[:create_disposition] = 'CREATE_IF_NEEDED'
+        load_configuration[:schema] = options[:schema] if options[:schema]
+        load_configuration[:source_format] = 'CSV'
+        load_configuration[:field_delimiter] = '\t'
+        load_configuration[:quote] = '"'
+        load_configuration[:destination_table] = @table_data.table_reference
+        job = Google::Apis::BigqueryV2::Job.new(
+          configuration: Google::Apis::BigqueryV2::JobConfiguration.new(
+            load: Google::Apis::BigqueryV2::JobConfigurationLoad.new(load_configuration)
+          )
+        )
+        job = @big_query_service.insert_job(@table_data.table_reference.project_id, job)
+        @logger.info(sprintf('Loading rows from %s to the table %s.%s', uri, @table_data.table_reference.dataset_id, @table_data.table_reference.table_id))
+        started = false
+        loop do
+          job = @big_query_service.get_job(@table_data.table_reference.project_id, job.job_reference.job_id)
+          if job.status && job.status.state == 'DONE'
+            if job.status.errors.nil? || job.status.errors.empty?
+              break
+            else
+              job.status.errors.each do |error|
+                message = %<Load error: "#{error.message}">
+                if error.location
+                  file, line, field = error.location.split('/').map { |s| s.split(':').last.strip }
+                  message << " at file #{file}, line #{line}"
+                  message << ", field #{field}" if field
+                end
+                @logger.debug(message)
+              end
+              raise job.status.error_result.message
+            end
+          else
+            state = job.status && job.status.state
+            if state == 'RUNNING' && !started
+              @logger.info('Loading started')
+              started = true
+            else
+              @logger.debug(sprintf('Waiting for job %s (status: %s)', job.job_reference.job_id.inspect, state ? state.inspect : 'unknown'))
+            end
+            @thread.sleep(poll_interval)
+          end
+        end
+        report_complete(job)
+        nil
+      end
+
+      private
+
+      DEFAULT_POLL_INTERVAL = 30
+
+      def report_complete(job)
+        statistics = job.statistics.load
+        input_size = statistics.input_file_bytes.to_f/2**30
+        output_size = statistics.output_bytes.to_f/2**30
+        @logger.info(sprintf('Loading complete, %.2f GiB loaded from %s files, %s rows created, table size %.2f GiB', input_size, statistics.input_files, statistics.output_rows, output_size))
+      end
+    end
+  end
+end

data/lib/bigshift/cli.rb

@@ -0,0 +1,186 @@
+require 'pg'
+require 'yaml'
+require 'json'
+require 'stringio'
+require 'logger'
+require 'optparse'
+require 'bigshift'
+
+module BigShift
+  class CliError < BigShiftError
+    attr_reader :details, :usage
+
+    def initialize(message, details, usage)
+      super(message)
+      @details = details
+      @usage = usage
+    end
+  end
+
+  class Cli
+    def initialize(argv, options={})
+      @argv = argv.dup
+      @factory_factory = options[:factory_factory] || Factory.method(:new)
+    end
+
+    def run
+      setup
+      unload
+      transfer
+      load
+      cleanup
+      nil
+    end
+
+    private
+
+    def setup
+      @config = parse_args(@argv)
+      @factory = @factory_factory.call(@config)
+    end
+
+    def unload
+      s3_uri = "s3://#{@config[:s3_bucket_name]}/#{s3_table_prefix}/"
+      @factory.redshift_unloader.unload_to(@config[:rs_table_name], s3_uri, allow_overwrite: true)
+    end
+
+    def transfer
+      description = "bigshift-#{@config[:rs_database_name]}-#{@config[:rs_table_name]}-#{Time.now.utc.strftime('%Y%m%dT%H%M')}"
+      @factory.cloud_storage_transfer.copy_to_cloud_storage(@config[:s3_bucket_name], "#{s3_table_prefix}/", @config[:cs_bucket_name], description: description, allow_overwrite: true)
+    end
+
+    def load
+      rs_table_schema = @factory.redshift_table_schema
+      bq_dataset = @factory.big_query_dataset
+      bq_table = bq_dataset.table(@config[:bq_table_id]) || bq_dataset.create_table(@config[:bq_table_id])
+      gcs_uri = "gs://#{@config[:cs_bucket_name]}/#{s3_table_prefix}/*"
+      bq_table.load(gcs_uri, schema: rs_table_schema.to_big_query, allow_overwrite: true)
+    end
+
+    def cleanup
+    end
+
+    ARGUMENTS = [
+      ['--gcp-credentials', 'PATH', :gcp_credentials_path, :required],
+      ['--aws-credentials', 'PATH', :aws_credentials_path, :required],
+      ['--rs-credentials', 'PATH', :rs_credentials_path, :required],
+      ['--rs-database', 'DB_NAME', :rs_database_name, :required],
+      ['--rs-table', 'TABLE_NAME', :rs_table_name, :required],
+      ['--bq-dataset', 'DATASET_ID', :bq_dataset_id, :required],
+      ['--bq-table', 'TABLE_ID', :bq_table_id, :required],
+      ['--s3-bucket', 'BUCKET_NAME', :s3_bucket_name, :required],
+      ['--s3-prefix', 'PREFIX', :s3_prefix, nil],
+      ['--cs-bucket', 'BUCKET_NAME', :cs_bucket_name, :required],
+    ]
+
+    def parse_args(argv)
+      config = {}
+      parser = OptionParser.new do |p|
+        ARGUMENTS.each do |flag, value_name, config_key, _|
+          p.on("#{flag} #{value_name}") { |v| config[config_key] = v }
+        end
+      end
+      config_errors = []
+      begin
+        parser.parse!(argv)
+      rescue OptionParser::InvalidOption => e
+        config_errors << e.message
+      end
+      %w[gcp aws rs].each do |prefix|
+        if (path = config["#{prefix}_credentials_path".to_sym]) && File.exist?(path)
+          config["#{prefix}_credentials".to_sym] = YAML.load(File.read(path))
+        elsif path && !File.exist?(path)
+          config_errors << sprintf('%s does not exist', path.inspect)
+        end
+      end
+      ARGUMENTS.each do |flag, _, config_key, required|
+        if !config.include?(config_key) && required
+          config_errors << "#{flag} is required"
+        end
+      end
+      unless config_errors.empty?
+        raise CliError.new('Configuration missing or malformed', config_errors, parser.to_s)
+      end
+      config
+    end
+
+    def s3_table_prefix
+      components = @config.values_at(:rs_database_name, :rs_table_name)
+      if (prefix = @config[:s3_prefix])
+        components.unshift(prefix)
+      end
+      File.join(*components)
+    end
+  end
+
+  class Factory
+    def initialize(config)
+      @config = config
+    end
+
+    def redshift_unloader
+      @redshift_unloader ||= RedshiftUnloader.new(rs_connection, aws_credentials, logger: logger)
+    end
+
+    def cloud_storage_transfer
+      @cloud_storage_transfer ||= CloudStorageTransfer.new(gcs_transfer_service, raw_gcp_credentials['project_id'], aws_credentials, logger: logger)
+    end
+
+    def redshift_table_schema
+      @redshift_table_schema ||= RedshiftTableSchema.new(@config[:rs_table_name], rs_connection)
+    end
+
+    def big_query_dataset
+      @big_query_dataset ||= BigQuery::Dataset.new(bq_service, raw_gcp_credentials['project_id'], @config[:bq_dataset_id], logger: logger)
+    end
+
+    private
+
+    def logger
+      @logger ||= Logger.new($stderr)
+    end
+
+    def rs_connection
+      @rs_connection ||= PG.connect(
+        @config[:rs_credentials]['host'],
+        @config[:rs_credentials]['port'],
+        nil,
+        nil,
+        @config[:rs_database_name],
+        @config[:rs_credentials]['username'],
+        @config[:rs_credentials]['password']
+      )
+    end
+
+    def gcs_transfer_service
+      @gcs_transfer_service ||= begin
+        s = Google::Apis::StoragetransferV1::StoragetransferService.new
+        s.authorization = gcp_credentials
+        s
+      end
+    end
+
+    def bq_service
+      @bq_service ||= begin
+        s = Google::Apis::BigqueryV2::BigqueryService.new
+        s.authorization = gcp_credentials
+        s
+      end
+    end
+
+    def aws_credentials
+      @config[:aws_credentials]
+    end
+
+    def raw_gcp_credentials
+      @config[:gcp_credentials]
+    end
+
+    def gcp_credentials
+      @gcp_credentials ||= Google::Auth::ServiceAccountCredentials.make_creds(
+        json_key_io: StringIO.new(JSON.dump(raw_gcp_credentials)),
+        scope: Google::Apis::StoragetransferV1::AUTH_CLOUD_PLATFORM
+      )
+    end
+  end
+end

data/lib/bigshift/cloud_storage_transfer.rb

@@ -0,0 +1,104 @@
+module BigShift
+  class CloudStorageTransfer
+    def initialize(storage_transfer_service, project_id, aws_credentials, options={})
+      @storage_transfer_service = storage_transfer_service
+      @project_id = project_id
+      @aws_credentials = aws_credentials
+      @clock = options[:clock] || Time
+      @thread = options[:thread] || Kernel
+      @logger = options[:logger] || NullLogger::INSTANCE
+    end
+
+    def copy_to_cloud_storage(s3_bucket, s3_path_prefix, cloud_storage_bucket, options={})
+      poll_interval = options[:poll_interval] || DEFAULT_POLL_INTERVAL
+      transfer_job = create_transfer_job(s3_bucket, s3_path_prefix, cloud_storage_bucket, options[:description], options[:allow_overwrite])
+      transfer_job = @storage_transfer_service.create_transfer_job(transfer_job)
+      @logger.info(sprintf('Transferring objects from s3://%s/%s to gs://%s/%s', s3_bucket, s3_path_prefix, cloud_storage_bucket, s3_path_prefix))
+      await_completion(transfer_job, poll_interval)
+      nil
+    end
+
+    private
+
+    DEFAULT_POLL_INTERVAL = 30
+
+    def create_transfer_job(s3_bucket, s3_path_prefix, cloud_storage_bucket, description, allow_overwrite)
+      now = @clock.now.utc
+      Google::Apis::StoragetransferV1::TransferJob.new(
+        description: description,
+        project_id: @project_id,
+        status: 'ENABLED',
+        schedule: Google::Apis::StoragetransferV1::Schedule.new(
+          schedule_start_date: Google::Apis::StoragetransferV1::Date.new(year: now.year, month: now.month, day: now.day),
+          schedule_end_date: Google::Apis::StoragetransferV1::Date.new(year: now.year, month: now.month, day: now.day),
+          start_time_of_day: Google::Apis::StoragetransferV1::TimeOfDay.new(hours: now.hour, minutes: now.min + 1)
+        ),
+        transfer_spec: Google::Apis::StoragetransferV1::TransferSpec.new(
+          aws_s3_data_source: Google::Apis::StoragetransferV1::AwsS3Data.new(
+            bucket_name: s3_bucket,
+            aws_access_key: Google::Apis::StoragetransferV1::AwsAccessKey.new(
+              access_key_id: @aws_credentials['aws_access_key_id'],
+              secret_access_key: @aws_credentials['aws_secret_access_key'],
+            )
+          ),
+          gcs_data_sink: Google::Apis::StoragetransferV1::GcsData.new(
+            bucket_name: cloud_storage_bucket
+          ),
+          object_conditions: Google::Apis::StoragetransferV1::ObjectConditions.new(
+            include_prefixes: [s3_path_prefix]
+          ),
+          transfer_options: Google::Apis::StoragetransferV1::TransferOptions.new(
+            overwrite_objects_already_existing_in_sink: !!allow_overwrite
+          )
+        )
+      )
+    end
+
+    def await_completion(transfer_job, poll_interval)
+      started = false
+      loop do
+        operation = nil
+        failures = 0
+        begin
+          operations_response = @storage_transfer_service.list_transfer_operations('transferOperations', filter: JSON.dump({project_id: @project_id, job_names: [transfer_job.name]}))
+          operation = operations_response.operations && operations_response.operations.first
+        rescue Google::Apis::ServerError => e
+          failures += 1
+          if failures < 5
+            @logger.debug(sprintf('Error while waiting for job %s, will retry: %s (%s)', transfer_job.name.inspect, e.message.inspect, e.class.name))
+            @thread.sleep(poll_interval)
+            retry
+          else
+            raise sprintf('Transfer failed: %s (%s)', e.message.inspect, e.class.name)
+          end
+        end
+        if operation && operation.done?
+          handle_completion(transfer_job, operation)
+          break
+        else
+          status = operation && operation.metadata && operation.metadata['status']
+          if status == 'IN_PROGRESS' && !started
+            @logger.info(sprintf('Transfer %s started', transfer_job.description))
+            started = true
+          else
+            @logger.debug(sprintf('Waiting for job %s (name: %s, status: %s)', transfer_job.description.inspect, transfer_job.name.inspect, status ? status.inspect : 'unknown'))
+          end
+          @thread.sleep(poll_interval)
+        end
+      end
+    end
+
+    def handle_completion(transfer_job, operation)
+      if operation.metadata['status'] == 'FAILED'
+        raise 'Transfer failed'
+      else
+        message = sprintf('Transfer %s complete', transfer_job.description)
+        if (counters = operation.metadata['counters'])
+          size_in_gib = counters['bytesCopiedToSink'].to_f / 2**30
+          message << sprintf(', %s objects and %.2f GiB copied', counters['objectsCopiedToSink'], size_in_gib)
+        end
+        @logger.info(message)
+      end
+    end
+  end
+end

data/lib/bigshift/redshift_table_schema.rb

@@ -0,0 +1,87 @@
+module BigShift
+  class RedshiftTableSchema
+    def initialize(table_name, redshift_connection)
+      @table_name = table_name
+      @redshift_connection = redshift_connection
+    end
+
+    def columns
+      @columns ||= begin
+        rows = @redshift_connection.exec_params(%|SELECT "column", "type", "notnull" FROM "pg_table_def" WHERE "schemaname" = 'public' AND "tablename" = $1|, [@table_name])
+        if rows.count == 0
+          raise sprintf('Table not found: %s', @table_name.inspect)
+        else
+          columns = rows.map do |row|
+            name = row['column']
+            type = row['type']
+            nullable = row['notnull'] == 'f'
+            Column.new(name, type, nullable)
+          end
+          columns.sort_by!(&:name)
+          columns
+        end
+      end
+    end
+
+    def to_big_query
+      Google::Apis::BigqueryV2::TableSchema.new(fields: columns.map(&:to_big_query))
+    end
+
+    class Column
+      attr_reader :name, :type
+
+      def initialize(name, type, nullable)
+        @name = name
+        @type = type
+        @nullable = nullable
+      end
+
+      def nullable?
+        @nullable
+      end
+
+      def to_big_query
+        Google::Apis::BigqueryV2::TableFieldSchema.new(
+          name: @name,
+          type: big_query_type,
+          mode: @nullable ? 'NULLABLE' : 'REQUIRED'
+        )
+      end
+
+      def to_sql
+        case @type
+        when /^numeric/, /int/, /^double/, 'real'
+          sprintf('"%s"', @name)
+        when /^character/
+          sprintf(%q<('"' || REPLACE(REPLACE(REPLACE("%s", '"', '""'), '\\n', '\\\\n'), '\\r', '\\\\r') || '"')>, @name)
+        when /^timestamp/
+          sprintf('(EXTRACT(epoch FROM "%s") + EXTRACT(milliseconds FROM "%s")/1000.0)', @name, @name)
+        when 'date'
+          sprintf(%q<(TO_CHAR("%s", 'YYYY-MM-DD'))>, @name)
+        when 'boolean'
+          if nullable?
+            sprintf('(CASE WHEN "%s" IS NULL THEN NULL WHEN "%s" THEN 1 ELSE 0 END)', @name, @name)
+          else
+            sprintf('(CASE WHEN "%s" THEN 1 ELSE 0 END)', @name)
+          end
+        else
+          raise sprintf('Unsupported column type: %s', type.inspect)
+        end
+      end
+
+      private
+
+      def big_query_type
+        case @type
+        when /^character/, /^numeric/, 'date' then 'STRING'
+        when /^timestamp/ then 'TIMESTAMP'
+        when /int/ then 'INTEGER'
+        when 'boolean' then 'BOOLEAN'
+        when /^double/, 'real' then 'FLOAT'
+        else
+          raise sprintf('Unsupported column type: %s', type.inspect)
+        end
+      end
+    end
+  end
+end

data/lib/bigshift/redshift_unloader.rb

@@ -0,0 +1,26 @@
+module BigShift
+  class RedshiftUnloader
+    def initialize(redshift_connection, aws_credentials, options={})
+      @redshift_connection = redshift_connection
+      @aws_credentials = aws_credentials
+      @logger = options[:logger] || NullLogger::INSTANCE
+    end
+
+    def unload_to(table_name, s3_uri, options={})
+      table_schema = RedshiftTableSchema.new(table_name, @redshift_connection)
+      credentials = @aws_credentials.map { |pair| pair.join('=') }.join(';')
+      select_sql = 'SELECT '
+      select_sql << table_schema.columns.map(&:to_sql).join(', ')
+      select_sql << %Q< FROM "#{table_name}">
+      select_sql.gsub!('\'') { |s| '\\\'' }
+      unload_sql = %Q<UNLOAD ('#{select_sql}')>
+      unload_sql << %Q< TO '#{s3_uri}'>
+      unload_sql << %Q< CREDENTIALS '#{credentials}'>
+      unload_sql << %q< DELIMITER '\t'>
+      unload_sql << %q< ALLOWOVERWRITE> if options[:allow_overwrite]
+      @logger.info(sprintf('Unloading Redshift table %s to %s', table_name, s3_uri))
+      @redshift_connection.exec(unload_sql)
+      @logger.info(sprintf('Unload of %s complete', table_name))
+    end
+  end
+end

data/lib/bigshift/version.rb

@@ -0,0 +1,3 @@
+module BigShift
+  VERSION = '0.1.1'.freeze
+end

metadata

@@ -0,0 +1,103 @@
+--- !ruby/object:Gem::Specification
+name: bigshift
+version: !ruby/object:Gem::Version
+  version: 0.1.1
+platform: ruby
+authors:
+- Theo Hultberg
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2016-04-08 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: pg
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: google-api-client
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.9'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.9'
+- !ruby/object:Gem::Dependency
+  name: googleauth
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: |-
+  BigShift is a tool for moving tables from Redshift
+                         to BigQuery. It will create a table in BigQuery with
+                         a schema that matches the Redshift table, dump the
+                         data to S3, transfer it to GCS and finally load it
+                         into the BigQuery table.
+email:
+- theo@iconara.net
+executables:
+- bigshift
+extensions: []
+extra_rdoc_files: []
+files:
+- LICENSE.txt
+- README.md
+- bin/bigshift
+- lib/bigshift.rb
+- lib/bigshift/big_query/dataset.rb
+- lib/bigshift/big_query/table.rb
+- lib/bigshift/cli.rb
+- lib/bigshift/cloud_storage_transfer.rb
+- lib/bigshift/redshift_table_schema.rb
+- lib/bigshift/redshift_unloader.rb
+- lib/bigshift/version.rb
+homepage: http://github.com/iconara/bigshift
+licenses:
+- BSD-3-Clause
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 1.9.3
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.8
+signing_key: 
+specification_version: 4
+summary: A tool for moving tables from Redshift to BigQuery
+test_files: []