redshift_extractor 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: aa8e751f08bdd4b304276a2c3689340736df59cc
-  data.tar.gz: 1e81e2a83c9a9f753e5a72a46656fe7b51ccc8fc
+  metadata.gz: d6b1a776fcff84a096fd9e510d1774fcd006191d
+  data.tar.gz: 0a69e8eb4ad78653b47a0151c9e6bd1d3389dbeb
 SHA512:
-  metadata.gz: eb63d429a7eb62e67ce033b6b33462978801cb98d9c6cab2f2fbf17c783313ca68928b9517adfe47cd6279882bb627ef859e16c810d47cee0a8b116f2507d79c
-  data.tar.gz: 4dfc53bdfc08aa9fb03d6736f80b9e100369a25a8a0273d95d92b6c8f0848c251d8a53a2f248c17f85327f345c4902ecc33ae93e40daf7ae0fee7844f3699ecd
+  metadata.gz: 32c29672a4b5ed08a44d56ccdd6e4447b0cddf48f1a5b534b0102ac6235cd4c3e927ae8bd9acfaf262f856bbef1a59131902122350d0c77329d65d1a92685bb1
+  data.tar.gz: d2ef768e4d19c90b0cb71617e019ba5365e05a65b2ad5f7f8e8d1acf49d4315cbe27d46518b1d7d25ed13c605d4f2b6028ca5df7f341a3079d87138ac3c392a4

data/README.md

@@ -1,40 +1,85 @@
 # RedshiftExtractor
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/redshift_extractor`. To experiment with that code, run `bin/console` for an interactive prompt.
+redshift_extractor moves data from one Amazon Redshift cluster to another.  Here is how it works:
 
-TODO: Delete this and the text above, and describe your gem
+- Source database
 
-## Installation
+  1. [UNLOAD](http://docs.aws.amazon.com/redshift/latest/dg/r_UNLOAD.html) - runs a SELECT query and exports the results to CSV files in S3.
 
-Add this line to your application's Gemfile:
+- Destination database
+
+  2. Drop - Drops a database table (the table in the destination database where the data will be stored).
+
+  3. Create - Creates a database table.
+
+  4. [COPY](http://docs.aws.amazon.com/redshift/latest/dg/r_COPY.html) - Loads data from S3 into a Redshift database.
+
+One database connection is established with the source database to UNLOAD the data to S3.  After the data is UNLOADed, a second database connection is establed with the destination database to drop/create the database table that will store the data.  The final step is to COPY the data from the S3 files to the destination table.
+
+## Running the Code
+
+The RedshiftExtractor::Extractor class is instantiated with a long hash of arguments.
 
 ```ruby
-gem 'redshift_extractor'
+args = {
+  database_config_source: "database_config_source",
+  database_config_destination: "database_config_destination",
+  unload_s3_destination: "unload_s3_destination",
+  unload_select_sql: "unload_select_sql",
+  table_name: "table_name",
+  create_sql: "create_sql",
+  copy_data_source: "copy_data_source",
+  aws_access_key_id: "aws_access_key_id",
+  aws_secret_access_key: "aws_secret_access_key"
+}
+
+extractor = RedshiftExtractor::Extractor.new(args)
+extractor.run
 ```
 
-And then execute:
+Here is a description of the parameters:
 
-    $ bundle
+- database_config_source: A hash that's acceptable for the [Ruby Postgres gem](https://bitbucket.org/ged/ruby-pg/wiki/Home).  Here's an example:
+
+```ruby
+{
+  dbname: "db_name",
+  user: "username",
+  password: "password",
+  host: "host",
+  sslmode: 'require',
+  port: 5439
+}
+```
+
+- unload_s3_destination: A S3 path, something like `"s3://bucket_name/something_else/"`
 
-Or install it yourself as:
+- unload_select_sql: A SQL SELECT query that will be run on the source table
 
-    $ gem install redshift_extractor
+- table_name: The table that will be dropped, recreated, and populated with data from the COPY command
 
-## Usage
+- create_sql: The SQL that creates the table_name table (this SQL is run to recreate the table in the step above)
 
-TODO: Write usage instructions here
+- copy_data_source: This is typically `"#{unload_s3_destination}manifest"`.  The UNLOAD command automatically creates a manifest file that can be used by the COPY command to load the data.
 
-## Development
+- aws_keys: The keys you get from AWS.
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+## Installation
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'redshift_extractor'
+```
+
+And then execute:
+
+    $ bundle
 
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/MrPowers/redshift_extractor.
 
-
 ## License
 
 The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).

data/lib/redshift_extractor/copy.rb

@@ -10,7 +10,9 @@ module RedshiftExtractor; class Copy
   end
 
   def copy_sql
-    "copy #{table_name} from '#{data_source}' credentials '#{credentials}' manifest dateformat 'auto' timeformat 'auto' blanksasnull emptyasnull escape gzip removequotes delimiter '|';"
+    "copy #{table_name} from '#{data_source}'"\
+    " credentials '#{credentials}'"\
+    " manifest dateformat 'auto' timeformat 'auto' blanksasnull emptyasnull escape gzip removequotes delimiter '|';"
   end
 
   def credentials

data/lib/redshift_extractor/extractor.rb

@@ -30,8 +30,11 @@ module RedshiftExtractor; class Extractor
     source_connection.exec(unloader.unload_sql)
   end
 
+  def dropper
+    Drop.new(table_name: config.table_name)
+  end
+
   def drop
-    dropper = Drop.new(table_name: config.table_name)
     destination_connection.exec(dropper.drop_sql)
   end
 
@@ -39,13 +42,16 @@ module RedshiftExtractor; class Extractor
     destination_connection.exec(config.create_sql)
   end
 
-  def copy
-    copier = Copy.new(
+  def copier
+    Copy.new(
       aws_access_key_id: config.aws_access_key_id,
       aws_secret_access_key: config.aws_secret_access_key,
       data_source: config.copy_data_source,
       table_name: config.table_name
     )
+  end
+
+  def copy
     destination_connection.exec(copier.copy_sql)
   end
 

data/lib/redshift_extractor/unload.rb

@@ -10,7 +10,8 @@ module RedshiftExtractor; class Unload
   end
 
   def unload_sql
-    "UNLOAD('#{escaped_extract_sql}') to '#{s3_destination}' CREDENTIALS '#{credentials}' MANIFEST GZIP ADDQUOTES ESCAPE;"
+    "UNLOAD('#{escaped_extract_sql}') to '#{s3_destination}'"\
+    " CREDENTIALS '#{credentials}' MANIFEST GZIP ADDQUOTES ESCAPE;"
   end
 
   def escaped_extract_sql

data/lib/redshift_extractor/version.rb

@@ -1,3 +1,3 @@
 module RedshiftExtractor
-  VERSION = "0.1.0"
+  VERSION = "0.2.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: redshift_extractor
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - MrPowers
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2015-10-27 00:00:00.000000000 Z
+date: 2015-10-31 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler