flare-up 0.8 → 0.9

checksums.yaml

@@ -1,15 +1,15 @@
 ---
 !binary "U0hBMQ==":
   metadata.gz: !binary |-
-    NzFmZDVjOWIyZmE4MDcxODhhM2E1YjhmNmE3YmNiODA2NWZhMjhkMg==
+    MWI2ZDkyYjVjOTBlMWMyYjI2NzM3NjNhNThkYzFlM2Q5OWYxNDQyNw==
   data.tar.gz: !binary |-
-    Y2ZmYTExMTRhMWUxYTdlNDQ4NjA5ZmRmODJiMjQxMDdmNjQ2ZGQyNg==
+    OTU0NTBiOGExMTA4MjhjOWViNzllMjUwMWU3YjU5NTcyODVlMmFhZA==
 SHA512:
   metadata.gz: !binary |-
-    MGY5YjZjNjI1ZWExZTRhODMyNWU2MzgzMGUyZTQ4NjJjNTk5MjYxNTliNzcw
-    N2M0ZDY0NWZlMmZjZWZiYzc2NTE3NjA3OTRjNTAxNDQ1OGYyMzc3MDFlOWMw
-    NzhlMjlmNDBkMjQ3ZGQ2ZjZhMGFhOTE1NmYyMDZjMjg0NDI5NjY=
+    MTM0MDVlMzAxOGUxOGQzNTRkNTQwMDBmNTQ4ZmU2MjY0OTRmZTI5YjY3OTZi
+    ZTBiYTUxNjk5YjZiYTI5ZDRkYjI5NDljOWY0YmI2NjBkYmZiMjFkZDgyZWVl
+    YmNmMjZjOWM0OWZmNmE0ZjZjN2E4OTlhZTNkOGFjY2E3MDY5MWY=
   data.tar.gz: !binary |-
-    MDFjZTI1MDU0OGE2OTcyYmMwOGM1OWIyYWJiYWJmNjFmMGVlYWM4ZWYxMjM2
-    YzIxMmEzMmZmYWY0OTg5YTJlMDRjYjJiMWM3NDExMTUxMGM1OGMxNGE2ZmRi
-    ZDhkNmM1ZmJjOTYyYmIxZDE3NmNkM2ZiMDFmOWMzZDU5MjJkNWU=
+    OWJkODg5ZmMyOGRjNDljZGIwNmE3MDdlMWMyMjUzZTBhY2Q5YjQ4MDNiN2Iw
+    MTUxZWVjZTE4YmIyZDY4ZjU1ZGQ2ZjEzZTgzMGFlYmU4OTY3NmIyZGVhNWNj
+    YmNkMDlhZmNhMzAzMzg1NmZjZjM4MWIyOGI1OTk1MDY1MGFmMDE=

data/.gitignore

@@ -1,3 +1,4 @@
 pkg/
 .DS_Store
 .env
+makefile

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    flare-up (0.8)
+    flare-up (0.9)
       pg (~> 0.17)
       thor (~> 0.19)
 

data/README.md

@@ -1,11 +1,11 @@
 ## Overview
-```Flare-up``` provides a wrapper around the Redshift [```COPY```](http://docs.aws.amazon.com/redshift/latest/dg/r_COPY.html) command for scriptability, allowing you to issue the command directly from the CLI.  Much of the code is concerned with simplifying constructing the COPY command and providing easy access to the errors that may result from import.
+```Flare-up``` provides a wrapper around the Redshift commands [```CREATE TABLE```](http://docs.aws.amazon.com/redshift/latest/dg/r_CREATE_TABLE_NEW.html), [```COPY```](http://docs.aws.amazon.com/redshift/latest/dg/r_COPY.html), and [```DROP TABLE```](http://docs.aws.amazon.com/redshift/latest/dg/r_DROP_TABLE.html)  for scriptability, allowing you to issue the commands directly from the CLI.  Much of the code is concerned with simplifying constructing the COPY command and providing easy access to the errors that may result from import.
 
 ## Why?
 
 Redshift prefers a bulk COPY operation over indidivual INSERTs which Redshift is not optimized for, and Amazon does not recommend it as a strategy for loading.  COPY is a SQL command, not something issued via the AWS Redshift REST API, meaning you need a SQL connection to your Redshift instance to bulk load data.
 
-The astute consumer of the AWS toolchain will note that [Data Pipeline](http://aws.amazon.com/datapipeline/) is one way this import may be completed however, we use Azkaban and the only thing worse one than one job flow control tool is two job flow control tools :)
+The astute consumer of the AWS toolchain will note that [Data Pipeline](http://aws.amazon.com/datapipeline/) is one way this import may be completed however, we use Azkaban and the only thing worse than one job flow control tool is two job flow control tools :)
 
 Additionally, access to COPY errors is a bit cumbersome.  On failure, Redshift populates the ```stl_load_errors``` table which inherently must be accessed via SQL.  Flare-up will pretty print any errors that occur during import so that you may examine your logs rather than establishing a connection to Redshift to understand what went wrong.
 
@@ -19,10 +19,12 @@ The `pg` gem is a dependency (required to issue SQL commands to Redshift) and wi
 
 ## Syntax
 
-Available via `flare-up help copy`.
+Available via `flare-up help <cmd>` where `<cmd>` can be replaced with `create_table`, `copy`, or `drop_table`.
 
 While we'd prefer if everyone stored configuration variables (esp. credentials) as environment variables (re: [Twelve-Factor App](http://12factor.net/)), it can be a pain to export variables when you're testing a tool and as such, we support specifying all of these on the command-line.
 
+### COPY
+
 ```
 Usage:
   flare-up copy DATA_SOURCE REDSHIFT_ENDPOINT DATABASE TABLE
@@ -38,9 +40,49 @@ Options:
                                                # Default: true
 ```
 
+### CREATE TABLE
+
+```
+Usage:
+  flare-up create_table REDSHIFT_ENDPOINT DATABASE TABLE
+
+Options:
+  [--column-list=COLUMN_LIST]                  # Required. A space-separated list of columns with their data-types, enclose "IN QUOTES"
+  [--redshift-username=REDSHIFT_USERNAME]      # Required unless ENV['REDSHIFT_USERNAME'] is set.
+  [--redshift-password=REDSHIFT_PASSWORD]      # Required unless ENV['REDSHIFT_PASSWORD'] is set.
+  [--colorize-output], [--no-colorize-output]  # Should Flare-up colorize its output?
+                                               # Default: true
+```
+
+### DROP TABLE
+
+```
+Usage:
+  flare-up drop_table REDSHIFT_ENDPOINT DATABASE TABLE
+
+Options:
+  [--redshift-username=REDSHIFT_USERNAME]      # Required unless ENV['REDSHIFT_USERNAME'] is set.
+  [--redshift-password=REDSHIFT_PASSWORD]      # Required unless ENV['REDSHIFT_PASSWORD'] is set.
+  [--colorize-output], [--no-colorize-output]  # Should Flare-up colorize its output?
+                                               # Default: true
+```
+
 ## Sample Usage
 
-Note that this example assumes you have credentials set as environment variables.
+Note that these examples assume you have credentials set as environment variables.
+
+### CREATE TABLE
+
+```
+> flare-up                                                      \
+    create_table                                                \
+    flare-up-test.cskjnp4xvaje.us-west-2.redshift.amazonaws.com \
+    dev                                                         \
+    hearthstone_cards                                           \
+    --column-list "id char(24) name varchar(2000)"
+```
+
+### COPY
 
 ```
 > flare-up                                                      \
@@ -50,5 +92,17 @@ Note that this example assumes you have credentials set as environment variables
     dev                                                         \
     hearthstone_cards                                           \
     --column-list name cost attack health description           \
-    --copy-options "REGION 'us-east-1' CSV"
+    --copy-options "REGION 'us-east-1' CSV IGNOREHEADER 1"
+```
+
+- The handy `IGNOREHEADER 1` option ignores the first line of field names  in the csv file.
+
+### DROP TABLE
+
+```
+> flare-up                                                      \
+    drop_table                                                  \
+    flare-up-test.cskjnp4xvaje.us-west-2.redshift.amazonaws.com \
+    dev                                                         \
+    hearthstone_cards
 ```

data/flare-up.gemspec

@@ -7,8 +7,8 @@ Gem::Specification.new do |s|
   s.platform    = Gem::Platform::RUBY
   s.authors     = ['Robert Slifka']
   s.homepage    = 'http://www.github.com/sharethrough/flare-up'
-  s.summary     = %q{Command-line access to bulk data loading via Redshift's COPY command.}
-  s.description = %q{Flare-up makes Redshift COPY scriptable by providing CLI access to the Redshift COPY command, with handy access to pretty printed errors as well.}
+  s.summary     = %q{Command-line access to bulk data loading via Redshift's CREATE TABLE, COPY, and DROP TABLE commands.}
+  s.description = %q{Flare-up makes Redshift COPY scriptable by providing CLI access to the Redshift COPY command, with handy access to pretty printed errors as well. It also includes the CREATE TABLE and DROP TABLE commands.}
 
   s.add_dependency('pg', '~> 0.17')
   s.add_dependency('thor', '~> 0.19')

data/lib/flare_up.rb

@@ -11,7 +11,10 @@ require 'flare_up/emitter'
 require 'flare_up/connection'
 require 'flare_up/stl_load_error'
 require 'flare_up/stl_load_error_fetcher'
-require 'flare_up/copy_command'
+require 'flare_up/command/base'
+require 'flare_up/command/copy'
+require 'flare_up/command/create_table'
+require 'flare_up/command/drop_table'
 
 require 'flare_up/boot'
 require 'flare_up/cli'

data/lib/flare_up/boot.rb

@@ -3,9 +3,9 @@ module FlareUp
   class Boot
 
     # TODO: This control flow is untested and too procedural
-    def self.boot
+    def self.boot(command)
       conn = create_connection
-      copy = create_copy_command
+      cmd = create_command(command)
 
       begin
         trap('SIGINT') do
@@ -19,12 +19,12 @@ module FlareUp
           CLI.bailout(1)
         end
 
-        Emitter.info("Executing command: #{copy.get_command}")
-        handle_load_errors(copy.execute(conn))
+        Emitter.info("Executing command: #{cmd.get_command}")
+        handle_load_errors(cmd.execute(conn))
       rescue ConnectionError => e
         Emitter.error(e.message)
         CLI.bailout(1)
-      rescue CopyCommandError => e
+      rescue CommandError => e
         Emitter.error(e.message)
         CLI.bailout(1)
       end
@@ -41,18 +41,18 @@ module FlareUp
     end
     private_class_method :create_connection
 
-    def self.create_copy_command
-      copy = FlareUp::CopyCommand.new(
+    def self.create_command(klass)
+      cmd = klass.new(
         OptionStore.get(:table),
         OptionStore.get(:data_source),
         OptionStore.get(:aws_access_key),
         OptionStore.get(:aws_secret_key)
       )
-      copy.columns = OptionStore.get(:column_list) if OptionStore.get(:column_list)
-      copy.options = OptionStore.get(:copy_options) if OptionStore.get(:copy_options)
-      copy
+      cmd.columns = OptionStore.get(:column_list) if OptionStore.get(:column_list)
+      cmd.options = OptionStore.get(:copy_options) if OptionStore.get(:copy_options)
+      cmd
     end
-    private_class_method :create_copy_command
+    private_class_method :create_command
 
     # TODO: Backfill tests
     def self.handle_load_errors(stl_load_errors)

data/lib/flare_up/cli.rb

@@ -2,44 +2,99 @@ module FlareUp
 
   class CLI < Thor
 
+    ### shared methods and variables
+
+    no_commands {
+      def command_setup(data_source, endpoint, database_name, table_name)
+        boot_options = {
+          :data_source => data_source,
+          :redshift_endpoint => endpoint,
+          :database => database_name,
+          :table => table_name
+        }
+        options.each { |k, v| boot_options[k.to_sym] = v }
+
+        begin
+          CLI.env_validator(boot_options, :aws_access_key, 'AWS_ACCESS_KEY_ID')
+          CLI.env_validator(boot_options, :aws_secret_key, 'AWS_SECRET_ACCESS_KEY')
+          CLI.env_validator(boot_options, :redshift_username, 'REDSHIFT_USERNAME')
+          CLI.env_validator(boot_options, :redshift_password, 'REDSHIFT_PASSWORD')
+        rescue ArgumentError => e
+          Emitter.error(e.message)
+          CLI.bailout(1)
+        end
+
+        OptionStore.store_options(boot_options)
+      end
+
+      def no_datasource_command(endpoint, database_name, table_name)
+        command_setup(nil, endpoint, database_name, table_name)
+        command = CLI.command_formatter __callee__
+        Boot.boot command
+      end
+    }
+
+    all_shared_options = [
+      [:redshift_username, { :type => :string, :desc => "Required unless ENV['REDSHIFT_USERNAME'] is set." }],
+      [:redshift_password, { :type => :string, :desc => "Required unless ENV['REDSHIFT_PASSWORD'] is set." }],
+      [:colorize_output, { :type => :boolean, :desc => 'Should Flare-up colorize its output?', :default => true }]
+    ]
+    copy_options = [
+      [:aws_access_key, { :type => :string, :desc => "Required unless ENV['AWS_ACCESS_KEY_ID'] is set." }],
+      [:aws_secret_key, { :type => :string, :desc => "Required unless ENV['AWS_SECRET_ACCESS_KEY'] is set." }],
+      [:copy_options, { :type => :string, :desc => "Appended to the end of the command; enclose \"IN QUOTES\"" }]
+    ]
+
+    long_desc_footer = <<-LONGDESCFOOTER
+\nDocumentation for this version can be found at:\x5
+https://github.com/sharethrough/flare-up/blob/v#{FlareUp::VERSION}/README.md
+    LONGDESCFOOTER
+
+    ### copy command
+
     desc 'copy DATA_SOURCE REDSHIFT_ENDPOINT DATABASE TABLE', 'COPY data into REDSHIFT_ENDPOINT from DATA_SOURCE into DATABASE.TABLE'
     long_desc <<-LONGDESC
 `flare-up copy` executes the Redshift COPY command, loading data from\x5
 DATA_SOURCE into DATABASE_NAME.TABLE_NAME at REDSHIFT_ENDPOINT.
-
-Documentation for this version can be found at:\x5
-https://github.com/sharethrough/flare-up/blob/v#{FlareUp::VERSION}/README.md
+#{long_desc_footer}
     LONGDESC
-    option :aws_access_key, :type => :string, :desc => "Required unless ENV['AWS_ACCESS_KEY_ID'] is set."
-    option :aws_secret_key, :type => :string, :desc => "Required unless ENV['AWS_SECRET_ACCESS_KEY'] is set."
-    option :redshift_username, :type => :string, :desc => "Required unless ENV['REDSHIFT_USERNAME'] is set."
-    option :redshift_password, :type => :string, :desc => "Required unless ENV['REDSHIFT_PASSWORD'] is set."
     option :column_list, :type => :array, :desc => 'A space-separated list of columns, should your DATA_SOURCE require it'
-    option :copy_options, :type => :string, :desc => "Appended to the end of the COPY command; enclose \"IN QUOTES\""
-    option :colorize_output, :type => :boolean, :desc => 'Should Flare-up colorize its output?', :default => true
-
+    [*all_shared_options, *copy_options].each { |shared_options| method_option *shared_options }
     def copy(data_source, endpoint, database_name, table_name)
-      boot_options = {
-        :data_source => data_source,
-        :redshift_endpoint => endpoint,
-        :database => database_name,
-        :table => table_name
-      }
-      options.each { |k, v| boot_options[k.to_sym] = v }
-
-      begin
-        CLI.env_validator(boot_options, :aws_access_key, 'AWS_ACCESS_KEY_ID')
-        CLI.env_validator(boot_options, :aws_secret_key, 'AWS_SECRET_ACCESS_KEY')
-        CLI.env_validator(boot_options, :redshift_username, 'REDSHIFT_USERNAME')
-        CLI.env_validator(boot_options, :redshift_password, 'REDSHIFT_PASSWORD')
-      rescue ArgumentError => e
-        Emitter.error(e.message)
-        CLI.bailout(1)
-      end
+      command_setup(data_source, endpoint, database_name, table_name)
+      command = CLI.command_formatter __method__
+      Boot.boot command
+    end
+
+    ### create_table command
 
-      OptionStore.store_options(boot_options)
+    desc 'create_table REDSHIFT_ENDPOINT DATABASE TABLE', 'CREATE DATABASE.TABLE in REDSHIFT_ENDPOINT'
+    long_desc <<-LONGDESC
+`flare-up create_table` executes the Redshift CREATE TABLE command, creating a table named\x5
+TABLE in DATABASE_NAME at REDSHIFT_ENDPOINT, using the scmhema provided in --column-list.
+#{long_desc_footer}
+    LONGDESC
+    option :column_list, { :type => :string, :desc => "Required. A space-separated list of columns with their data-types, enclose \"IN QUOTES\"" }
+    [*all_shared_options].each { |shared_options| method_option *shared_options }
+    alias_method :create_table, :no_datasource_command
+
+    ### drop_table command
+
+    desc 'drop_table REDSHIFT_ENDPOINT DATABASE TABLE', 'DROP DATABASE.TABLE in REDSHIFT_ENDPOINT'
+    long_desc <<-LONGDESC
+`flare-up drop_table` executes the Redshift DROP TABLE command, removing the table named TABLE\x5
+in DATABASE_NAME at REDSHIFT_ENDPOINT.
+#{long_desc_footer}
+    LONGDESC
+    all_shared_options.each { |shared_options| method_option *shared_options }
+    alias_method :drop_table, :no_datasource_command
 
-      Boot.boot
+    # transforms the symbol method names to the corresponding class under
+    # the FlareUp::Command namespace
+    def self.command_formatter(sym)
+      name = sym.to_s.split('_').map(&:capitalize).join
+      # Ruby 1.9.3 cannot handle const_get for nested Classes (Ruby 2.0 can).
+      "FlareUp::Command::#{name}".split("::").reduce(Object) { |a, e| a.const_get e }
     end
 
     def self.env_validator(options, option_name, env_variable_name)

data/lib/flare_up/command/base.rb

@@ -0,0 +1,22 @@
+module FlareUp
+
+  class CommandError < StandardError
+  end
+  class DataSourceError < CommandError
+  end
+  class OtherZoneBucketError < CommandError
+  end
+  class SyntaxError < CommandError
+  end
+
+  module Command
+    class Base
+
+      attr_reader :table_name
+
+      def initialize(table_name, *args)
+        @table_name = table_name
+      end
+    end
+  end
+end

data/lib/flare_up/command/copy.rb

@@ -0,0 +1,63 @@
+module FlareUp
+  module Command
+    class Copy < Command::Base
+
+      attr_reader :data_source
+      attr_reader :aws_access_key_id
+      attr_reader :aws_secret_access_key
+      attr_accessor :options
+      attr_reader :columns
+
+      def initialize(table_name, data_source, aws_access_key_id, aws_secret_access_key)
+        @data_source = data_source
+        @aws_access_key_id = aws_access_key_id
+        @aws_secret_access_key = aws_secret_access_key
+        @options = ''
+        @columns = []
+        super
+      end
+
+      # http://docs.aws.amazon.com/redshift/latest/dg/r_COPY.html
+      def get_command
+        "COPY #{@table_name} #{get_columns} FROM '#{@data_source}' CREDENTIALS '#{get_credentials}' #{@options}"
+      end
+
+      def columns=(columns)
+        raise ArgumentError, 'Columns must be an array' unless columns.is_a?(Array)
+        @columns = columns
+      end
+
+      def execute(connection)
+        begin
+          connection.execute(get_command)
+          []
+        rescue PG::InternalError => e
+          case e.message
+            when /Check 'stl_load_errors' system table for details/
+              return STLLoadErrorFetcher.fetch_errors(connection)
+            when /The specified S3 prefix '.+' does not exist/
+              raise DataSourceError, "A data source with prefix '#{@data_source}' does not exist."
+            when /The bucket you are attempting to access must be addressed using the specified endpoint/
+              raise OtherZoneBucketError, "Your Redshift instance appears to be in a different zone than your S3 bucket.  Specify the \"REGION 'bucket-region'\" option."
+            when /PG::SyntaxError/
+              matches = /syntax error (.+) \(PG::SyntaxError\)/.match(e.message)
+              raise SyntaxError, "Syntax error in the COPY command: [#{matches[1]}]."
+            else
+              raise e
+          end
+        end
+      end
+
+      private
+
+      def get_columns
+        return '' if columns.empty?
+        "(#{@columns.join(', ').strip})"
+      end
+
+      def get_credentials
+        "aws_access_key_id=#{@aws_access_key_id};aws_secret_access_key=#{@aws_secret_access_key}"
+      end
+    end
+  end
+end

data/lib/flare_up/command/create_table.rb

@@ -0,0 +1,58 @@
+module FlareUp
+  module Command
+    class CreateTable < Command::Base
+
+      attr_reader :columns
+
+      def initialize(*args)
+        @columns = []
+        super
+      end
+
+
+      # http://docs.aws.amazon.com/redshift/latest/dg/r_CREATE_TABLE_NEW.html
+      def get_command
+        "CREATE TABLE #{@table_name} #{get_columns} #{@options}"
+      end
+
+      # a little different than copy... the columns argument will have parentheses
+      # (since it specifies a schema, where the datatypes may require parentheses)
+      # and will need to be enclosed in quotes, and therefore the argument arrives
+      # here as a single membered array. This method corrects this difference between
+      # copy and create table by splitting on spaces, so that the columns are stored
+      # in the same fashion between the two classes, though they arrive in different
+      # forms to this method.
+      def columns=(columns)
+        raise ArgumentError, 'Columns must be a string' unless columns.is_a?(String)
+        columns_separated = columns.split(' ')
+        raise ArgumentError, 'Columns must have a data type for each name' unless columns_separated.length % 2 == 0
+        @columns = columns_separated
+      end
+
+      def execute(connection)
+        begin
+          connection.execute(get_command)
+          []
+        rescue PG::InternalError => e
+          case e.message
+            when /Check 'stl_load_errors' system table for details/
+              return STLLoadErrorFetcher.fetch_errors(connection)
+            when /PG::SyntaxError/
+              matches = /syntax error (.+) \(PG::SyntaxError\)/.match(e.message)
+              raise SyntaxError, "Syntax error in the CREATE TABLE command: [#{matches[1]}]."
+            else
+              raise e
+          end
+        end
+      end
+
+      private
+
+      def get_columns
+        return '' if columns.empty?
+        name_type_pairs = @columns.each_slice(2).map { |name, type| "#{name} #{type}" }
+        "(#{name_type_pairs.join(', ').strip})"
+      end
+    end
+  end
+end

data/lib/flare_up/command/drop_table.rb

@@ -0,0 +1,28 @@
+module FlareUp
+  module Command
+    class DropTable < Command::Base
+
+      # http://docs.aws.amazon.com/redshift/latest/dg/r_DROP_TABLE.html
+      def get_command
+        "DROP TABLE #{@table_name} #{@options}"
+      end
+
+      def execute(connection)
+        begin
+          connection.execute(get_command)
+          []
+        rescue PG::InternalError => e
+          case e.message
+            when /Check 'stl_load_errors' system table for details/
+              return STLLoadErrorFetcher.fetch_errors(connection)
+            when /PG::SyntaxError/
+              matches = /syntax error (.+) \(PG::SyntaxError\)/.match(e.message)
+              raise SyntaxError, "Syntax error in the DROP TABLE command: [#{matches[1]}]."
+            else
+              raise e
+          end
+        end
+      end
+    end
+  end
+end

data/lib/flare_up/version.rb

@@ -1,3 +1,3 @@
 module FlareUp
-  VERSION = '0.8'
+  VERSION = '0.9'
 end

data/resources/load_hearthstone_cards.rb

@@ -7,7 +7,7 @@ data_source = 's3://slif-redshift/hearthstone_cards_short_list.csv'
 
 conn = FlareUp::Connection.new(host_name, db_name, ENV['REDSHIFT_USERNAME'], ENV['REDSHIFT_PASSWORD'])
 
-copy = FlareUp::CopyCommand.new(table_name, data_source, ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY'])
+copy = FlareUp::Command::Copy.new(table_name, data_source, ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY'])
 copy.columns = %w(name cost attack health description)
 copy.options = "REGION 'us-east-1' CSV"
 

data/spec/lib/flare_up/boot_spec.rb

@@ -2,7 +2,8 @@ describe FlareUp::Boot do
 
   describe '.boot' do
     let(:connection) { instance_double('FlareUp::Connection') }
-    let(:copy_command) { instance_double('FlareUp::CopyCommand') }
+    let(:copy_command) { instance_double('FlareUp::Command::Copy') }
+    let(:klass) { FlareUp::Command::Copy }
 
     before do
       allow(copy_command).to receive(:get_command)
@@ -11,7 +12,7 @@ describe FlareUp::Boot do
     context 'when there is an error connecting' do
 
       before do
-        expect(FlareUp::Boot).to receive(:create_copy_command).and_return(copy_command)
+        expect(FlareUp::Boot).to receive(:create_command).and_return(copy_command)
         expect(copy_command).to receive(:execute).and_raise(copy_command_error)
       end
 
@@ -19,7 +20,7 @@ describe FlareUp::Boot do
         let(:copy_command_error) { FlareUp::DataSourceError }
         it 'should handle the error' do
           expect(FlareUp::CLI).to receive(:bailout).with(1)
-          expect { FlareUp::Boot.boot }.not_to raise_error
+          expect { FlareUp::Boot.boot klass }.not_to raise_error
         end
       end
 
@@ -27,7 +28,7 @@ describe FlareUp::Boot do
         let(:copy_command_error) { FlareUp::OtherZoneBucketError }
         it 'should handle the error' do
           expect(FlareUp::CLI).to receive(:bailout).with(1)
-          expect { FlareUp::Boot.boot }.not_to raise_error
+          expect { FlareUp::Boot.boot klass }.not_to raise_error
         end
       end
 
@@ -35,7 +36,7 @@ describe FlareUp::Boot do
         let(:copy_command_error) { FlareUp::SyntaxError }
         it 'should handle the error' do
           expect(FlareUp::CLI).to receive(:bailout).with(1)
-          expect { FlareUp::Boot.boot }.not_to raise_error
+          expect { FlareUp::Boot.boot klass }.not_to raise_error
         end
       end
 
@@ -52,7 +53,7 @@ describe FlareUp::Boot do
         let(:connection_error) { FlareUp::HostUnknownOrInaccessibleError }
         it 'should handle the error' do
           expect(FlareUp::CLI).to receive(:bailout).with(1)
-          expect { FlareUp::Boot.boot }.not_to raise_error
+          expect { FlareUp::Boot.boot klass }.not_to raise_error
         end
       end
 
@@ -60,7 +61,7 @@ describe FlareUp::Boot do
         let(:connection_error) { FlareUp::TimeoutError }
         it 'should handle the error' do
           expect(FlareUp::CLI).to receive(:bailout).with(1)
-          expect { FlareUp::Boot.boot }.not_to raise_error
+          expect { FlareUp::Boot.boot klass }.not_to raise_error
         end
       end
 
@@ -68,7 +69,7 @@ describe FlareUp::Boot do
         let(:connection_error) { FlareUp::NoDatabaseError }
         it 'should handle the error' do
           expect(FlareUp::CLI).to receive(:bailout).with(1)
-          expect { FlareUp::Boot.boot }.not_to raise_error
+          expect { FlareUp::Boot.boot klass }.not_to raise_error
         end
       end
 
@@ -76,7 +77,7 @@ describe FlareUp::Boot do
         let(:connection_error) { FlareUp::AuthenticationError }
         it 'should handle the error' do
           expect(FlareUp::CLI).to receive(:bailout).with(1)
-          expect { FlareUp::Boot.boot }.not_to raise_error
+          expect { FlareUp::Boot.boot klass }.not_to raise_error
         end
       end
 
@@ -84,7 +85,7 @@ describe FlareUp::Boot do
         let(:connection_error) { FlareUp::UnknownError }
         it 'should handle the error' do
           expect(FlareUp::CLI).to receive(:bailout).with(1)
-          expect { FlareUp::Boot.boot }.not_to raise_error
+          expect { FlareUp::Boot.boot klass }.not_to raise_error
         end
       end
 
@@ -113,7 +114,7 @@ describe FlareUp::Boot do
     end
   end
 
-  describe '.create_copy_command' do
+  describe '.create_command' do
 
     before do
       FlareUp::OptionStore.store_options(
@@ -127,7 +128,7 @@ describe FlareUp::Boot do
     end
 
     it 'should create a proper copy command' do
-      command = FlareUp::Boot.send(:create_copy_command)
+      command = FlareUp::Boot.send(:create_command, FlareUp::Command::Copy)
       expect(command.table_name).to eq('TEST_TABLE')
       expect(command.data_source).to eq('TEST_DATA_SOURCE')
       expect(command.aws_access_key_id).to eq('TEST_ACCESS_KEY')
@@ -139,7 +140,7 @@ describe FlareUp::Boot do
         FlareUp::OptionStore.store_option(:column_list, ['c1'])
       end
       it 'should create a proper copy command' do
-        command = FlareUp::Boot.send(:create_copy_command)
+        command = FlareUp::Boot.send(:create_command, FlareUp::Command::Copy)
         expect(command.columns).to eq(['c1'])
       end
     end
@@ -149,7 +150,7 @@ describe FlareUp::Boot do
         FlareUp::OptionStore.store_option(:copy_options, '_')
       end
       it 'should create a proper copy command' do
-        command = FlareUp::Boot.send(:create_copy_command)
+        command = FlareUp::Boot.send(:create_command, FlareUp::Command::Copy)
         expect(command.options).to eq('_')
       end
     end

data/spec/lib/flare_up/{copy_command_spec.rb → command/copy_spec.rb}

@@ -1,7 +1,7 @@
-describe FlareUp::CopyCommand do
+describe FlareUp::Command::Copy do
 
   subject do
-    FlareUp::CopyCommand.new('TEST_TABLE_NAME', 'TEST_DATA_SOURCE', 'TEST_ACCESS_KEY', 'TEST_SECRET_KEY')
+    FlareUp::Command::Copy.new('TEST_TABLE_NAME', 'TEST_DATA_SOURCE', 'TEST_ACCESS_KEY', 'TEST_SECRET_KEY')
   end
 
   its(:table_name) { should == 'TEST_TABLE_NAME' }
@@ -116,7 +116,6 @@ describe FlareUp::CopyCommand do
             expect { subject.execute(conn) }.to raise_error(FlareUp::SyntaxError, 'Syntax error in the COPY command: [at or near "lmlkmlk3"].')
           end
         end
-
       end
 
       context 'when there was another type of error' do
@@ -126,9 +125,6 @@ describe FlareUp::CopyCommand do
           expect { subject.execute(conn) }.to raise_error(PG::ConnectionBad, '_')
         end
       end
-
     end
-
   end
-
-end
+end

data/spec/lib/flare_up/command/create_table_spec.rb

@@ -0,0 +1,117 @@
+describe FlareUp::Command::CreateTable do
+
+  subject do
+    FlareUp::Command::CreateTable.new('TEST_TABLE_NAME', nil, nil, nil)
+  end
+
+  its(:table_name) { should == 'TEST_TABLE_NAME' }
+  its(:columns) { should == [] }
+
+  describe '#get_command' do
+    context 'when no optional fields are provided' do
+      it 'should return a basic CREATE TABLE command' do
+        expect(subject.get_command).to eq("CREATE TABLE TEST_TABLE_NAME  ")
+      end
+    end
+
+    context 'when column names are provided' do
+      before do
+        subject.columns = 'column_name1 char(24) column_name2 varchar(2000)'
+      end
+      it 'should include the column names in the command' do
+        expect(subject.get_command).to start_with('CREATE TABLE TEST_TABLE_NAME (column_name1 char(24), column_name2 varchar(2000))')
+      end
+    end
+  end
+
+  describe '#columns=' do
+    context 'when a string' do
+      it 'should assign the property' do
+        subject.columns = 'column_name1 char(24) column_name2 varchar(2000)'
+        expect(subject.columns).to eq(['column_name1', 'char(24)', 'column_name2', 'varchar(2000)'])
+      end
+      it 'should assign an empty array for an empty string' do
+        subject.columns = ''
+        expect(subject.columns).to eq([])
+      end
+    end
+
+    context 'when not a string' do
+      it 'should not assign the property and be an error' do
+        subject.columns = 'column_name1 char(24)'
+        expect {
+          subject.columns = ['column_name_in_arr char(24)']
+        }.to raise_error(ArgumentError)
+        expect(subject.columns).to eq(['column_name1', 'char(24)'])
+      end
+    end
+
+    context 'when not a string of name, type pairs' do
+      it 'should not assign the property and be an error' do
+        subject.columns = 'column_name1 char(24)'
+        expect {
+          subject.columns = 'column_name1'
+        }.to raise_error(ArgumentError)
+        expect(subject.columns).to eq(['column_name1', 'char(24)'])
+      end
+    end
+  end
+
+  describe '#execute' do
+
+    let(:conn) { instance_double('FlareUp::Connection') }
+
+    context 'when successful' do
+      before do
+        expect(conn).to receive(:execute)
+      end
+      it 'should do something' do
+        expect(subject.execute(conn)).to eq([])
+      end
+    end
+
+    context 'when unsuccessful' do
+
+      before do
+        expect(conn).to receive(:execute).and_raise(exception, message)
+      end
+
+      context 'when there was an internal error' do
+
+        let(:exception) { PG::InternalError }
+
+        context 'when there was an error loading' do
+          let(:message) { "Check 'stl_load_errors' system table for details" }
+          before do
+            allow(FlareUp::STLLoadErrorFetcher).to receive(:fetch_errors).and_return('FOO')
+          end
+          it 'should respond with a list of errors' do
+            expect(subject.execute(conn)).to eq('FOO')
+          end
+        end
+
+        context 'when there was another kind of internal error' do
+          let(:message) { '_' }
+          it 'should respond with a list of errors' do
+            expect { subject.execute(conn) }.to raise_error(PG::InternalError, '_')
+          end
+        end
+
+        context 'when there is a syntax error in the command' do
+          let(:message) { 'ERROR:  syntax error at or near "lmlkmlk3" (PG::SyntaxError)' }
+          it 'should be error' do
+            expect { subject.execute(conn) }.to raise_error(FlareUp::SyntaxError, 'Syntax error in the CREATE TABLE command: [at or near "lmlkmlk3"].')
+          end
+        end
+      end
+
+      context 'when there was another type of error' do
+        let(:exception) { PG::ConnectionBad }
+        let(:message) { '_' }
+        it 'should do something' do
+          expect { subject.execute(conn) }.to raise_error(PG::ConnectionBad, '_')
+        end
+      end
+    end
+  end
+end

data/spec/lib/flare_up/command/drop_table_spec.rb

@@ -0,0 +1,74 @@
+describe FlareUp::Command::DropTable do
+
+  subject do
+    FlareUp::Command::DropTable.new('TEST_TABLE_NAME', nil, nil, nil)
+  end
+
+  its(:table_name) { should == 'TEST_TABLE_NAME' }
+
+  describe '#get_command' do
+    context 'when no optional fields are provided' do
+      it 'should return a basic DROP TABLE command' do
+        expect(subject.get_command).to eq("DROP TABLE TEST_TABLE_NAME ")
+      end
+    end
+  end
+
+  describe '#execute' do
+
+    let(:conn) { instance_double('FlareUp::Connection') }
+
+    context 'when successful' do
+      before do
+        expect(conn).to receive(:execute)
+      end
+      it 'should do something' do
+        expect(subject.execute(conn)).to eq([])
+      end
+    end
+
+    context 'when unsuccessful' do
+
+      before do
+        expect(conn).to receive(:execute).and_raise(exception, message)
+      end
+
+      context 'when there was an internal error' do
+
+        let(:exception) { PG::InternalError }
+
+        context 'when there was an error loading' do
+          let(:message) { "Check 'stl_load_errors' system table for details" }
+          before do
+            allow(FlareUp::STLLoadErrorFetcher).to receive(:fetch_errors).and_return('FOO')
+          end
+          it 'should respond with a list of errors' do
+            expect(subject.execute(conn)).to eq('FOO')
+          end
+        end
+
+        context 'when there was another kind of internal error' do
+          let(:message) { '_' }
+          it 'should respond with a list of errors' do
+            expect { subject.execute(conn) }.to raise_error(PG::InternalError, '_')
+          end
+        end
+
+        context 'when there is a syntax error in the command' do
+          let(:message) { 'ERROR:  syntax error at or near "lmlkmlk3" (PG::SyntaxError)' }
+          it 'should be error' do
+            expect { subject.execute(conn) }.to raise_error(FlareUp::SyntaxError, 'Syntax error in the DROP TABLE command: [at or near "lmlkmlk3"].')
+          end
+        end
+      end
+
+      context 'when there was another type of error' do
+        let(:exception) { PG::ConnectionBad }
+        let(:message) { '_' }
+        it 'should do something' do
+          expect { subject.execute(conn) }.to raise_error(PG::ConnectionBad, '_')
+        end
+      end
+    end
+  end
+end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: flare-up
 version: !ruby/object:Gem::Version
-  version: '0.8'
+  version: '0.9'
 platform: ruby
 authors:
 - Robert Slifka
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-11-13 00:00:00.000000000 Z
+date: 2014-11-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: pg
@@ -81,7 +81,8 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '1.0'
 description: Flare-up makes Redshift COPY scriptable by providing CLI access to the
-  Redshift COPY command, with handy access to pretty printed errors as well.
+  Redshift COPY command, with handy access to pretty printed errors as well. It also
+  includes the CREATE TABLE and DROP TABLE commands.
 email: 
 executables:
 - flare-up
@@ -102,8 +103,11 @@ files:
 - lib/flare_up.rb
 - lib/flare_up/boot.rb
 - lib/flare_up/cli.rb
+- lib/flare_up/command/base.rb
+- lib/flare_up/command/copy.rb
+- lib/flare_up/command/create_table.rb
+- lib/flare_up/command/drop_table.rb
 - lib/flare_up/connection.rb
-- lib/flare_up/copy_command.rb
 - lib/flare_up/emitter.rb
 - lib/flare_up/env_wrap.rb
 - lib/flare_up/option_store.rb
@@ -117,8 +121,10 @@ files:
 - resources/test_schema.sql
 - spec/lib/flare_up/boot_spec.rb
 - spec/lib/flare_up/cli_spec.rb
+- spec/lib/flare_up/command/copy_spec.rb
+- spec/lib/flare_up/command/create_table_spec.rb
+- spec/lib/flare_up/command/drop_table_spec.rb
 - spec/lib/flare_up/connection_spec.rb
-- spec/lib/flare_up/copy_command_spec.rb
 - spec/lib/flare_up/emitter_spec.rb
 - spec/lib/flare_up/option_store_spec.rb
 - spec/lib/flare_up/stl_load_error_fetcher_spec.rb
@@ -146,12 +152,15 @@ rubyforge_project:
 rubygems_version: 2.2.2
 signing_key: 
 specification_version: 4
-summary: Command-line access to bulk data loading via Redshift's COPY command.
+summary: Command-line access to bulk data loading via Redshift's CREATE TABLE, COPY,
+  and DROP TABLE commands.
 test_files:
 - spec/lib/flare_up/boot_spec.rb
 - spec/lib/flare_up/cli_spec.rb
+- spec/lib/flare_up/command/copy_spec.rb
+- spec/lib/flare_up/command/create_table_spec.rb
+- spec/lib/flare_up/command/drop_table_spec.rb
 - spec/lib/flare_up/connection_spec.rb
-- spec/lib/flare_up/copy_command_spec.rb
 - spec/lib/flare_up/emitter_spec.rb
 - spec/lib/flare_up/option_store_spec.rb
 - spec/lib/flare_up/stl_load_error_fetcher_spec.rb

data/lib/flare_up/copy_command.rb

@@ -1,73 +0,0 @@
-module FlareUp
-
-  class CopyCommandError < StandardError
-  end
-  class DataSourceError < CopyCommandError
-  end
-  class OtherZoneBucketError < CopyCommandError
-  end
-  class SyntaxError < CopyCommandError
-  end
-
-  class CopyCommand
-
-    attr_reader :table_name
-    attr_reader :data_source
-    attr_reader :aws_access_key_id
-    attr_reader :aws_secret_access_key
-    attr_reader :columns
-    attr_accessor :options
-
-    def initialize(table_name, data_source, aws_access_key_id, aws_secret_access_key)
-      @table_name = table_name
-      @data_source = data_source
-      @aws_access_key_id = aws_access_key_id
-      @aws_secret_access_key = aws_secret_access_key
-      @columns = []
-      @options = ''
-    end
-
-    def get_command
-      "COPY #{@table_name} #{get_columns} FROM '#{@data_source}' CREDENTIALS '#{get_credentials}' #{@options}"
-    end
-
-    def columns=(columns)
-      raise ArgumentError, 'Columns must be an array' unless columns.is_a?(Array)
-      @columns = columns
-    end
-
-    def execute(connection)
-      begin
-        connection.execute(get_command)
-        []
-      rescue PG::InternalError => e
-        case e.message
-          when /Check 'stl_load_errors' system table for details/
-            return STLLoadErrorFetcher.fetch_errors(connection)
-          when /The specified S3 prefix '.+' does not exist/
-            raise DataSourceError, "A data source with prefix '#{@data_source}' does not exist."
-          when /The bucket you are attempting to access must be addressed using the specified endpoint/
-            raise OtherZoneBucketError, "Your Redshift instance appears to be in a different zone than your S3 bucket.  Specify the \"REGION 'bucket-region'\" option."
-          when /PG::SyntaxError/
-            matches = /syntax error (.+) \(PG::SyntaxError\)/.match(e.message)
-            raise SyntaxError, "Syntax error in the COPY command: [#{matches[1]}]."
-          else
-            raise e
-        end
-      end
-    end
-
-    private
-
-    def get_columns
-      return '' if columns.empty?
-      "(#{@columns.join(', ').strip})"
-    end
-
-    def get_credentials
-      "aws_access_key_id=#{@aws_access_key_id};aws_secret_access_key=#{@aws_secret_access_key}"
-    end
-
-  end
-
-end