elasticity 2.3.1 → 2.4

data/.travis.yml

@@ -4,8 +4,3 @@ rvm:
   - 1.9.2
   - 1.9.3
   - ree
-branches:
-  only:
-    - master
-    - travis-testing
-

data/HISTORY.md

@@ -1,8 +1,16 @@
+## 2.4 - September 1, 2012
+
++ ```SyncToS3``` added to enable one-way asset synchronization.
++ Generic bootstrap actions are now supported via ```BootstrapAction```.
++ If you have several Hadoop bootstrap actions (15 is the current EMR limit), store all of your Hadoop configuration options in a file, ship it up with ```SyncToS3``` and use the new ```HadoopFileBootstrapAction``` to point at that file.
++ If no parameters are passed to ```JobFlow.new```, it will use the standard AWS environment variables to lookup the access and secret keys - ```AWS_ACCESS_KEY_ID``` and ```AWS_SECRET_ACCESS_KEY```.
++ New dependencies: [fog](https://github.com/fog/fog) (S3 access), [fakefs](https://github.com/defunkt/fakefs) (filesystem stubbing - development only), [timecop](https://github.com/jtrupiano/timecop) (freezing and manipulating time - development only).
+
 ## 2.3.1 - August 23, 2012
 
 + Birthday release! ;)
 + Bumped the default version of Hadoop to 1.0.3.
-+ Amazon now requires the --hive-versions argument when installing Hive (thanks to Johannes Wuerbach).
++ Amazon now requires the ```--hive-versions``` argument when installing Hive (thanks to Johannes Wuerbach).
 + ```JobFlowStatus#master_public_dns_name``` is now available (thanks to Johannes Wuerbach).
 
 ## 2.3 - July 28, 2012

data/README.md

@@ -1,29 +1,29 @@
-Elasticity provides programmatic access to Amazon's Elastic Map Reduce service.  The aim is to conveniently map the EMR REST API calls to higher level operations that make working with job flows more productive and more enjoyable.
+Elasticity provides programmatic access to Amazon's Elastic Map Reduce service.  The aim is to conveniently abstract away the complex EMR REST API and make working with job flows more productive and more enjoyable.
 
 [![Build Status](https://secure.travis-ci.org/rslifka/elasticity.png)](http://travis-ci.org/rslifka/elasticity) REE, 1.8.7, 1.9.2, 1.9.3
 
 Elasticity provides two ways to access EMR:
 
 * **Indirectly through a JobFlow-based API**. This README discusses the Elasticity API.
-* **Directly through access to the EMR REST API**. The less-discussed hidden darkside... I use this to enable the Elasticity API though it is not documented save for RubyDoc available at the the RubyGems [auto-generated documentation site](http://rubydoc.info/gems/elasticity/frames).  Be forewarned: Making the calls directly requires that you understand how to structure EMR requests at the Amazon API level and from experience I can tell you there are more fun things you could be doing :)  Scroll to the end for more information on the Amazon API. 
+* **Directly through access to the EMR REST API**. The less-discussed hidden darkside... I use this to enable the Elasticity API.  RubyDoc can be found at the RubyGems [auto-generated documentation site](http://rubydoc.info/gems/elasticity/frames).  Be forewarned: Making the calls directly requires that you understand how to structure EMR requests at the Amazon API level and from experience I can tell you there are more fun things you could be doing :)  Scroll to the end for more information on the Amazon API.
 
 # Installation
 
-```ruby
-  gem install elasticity
+```
+gem install elasticity
 ```
 
 or in your Gemfile
 
-```ruby
-  gem 'elasticity', '~> 2.0'
+```
+gem 'elasticity', '~> 2.0'
 ```
 
 This will ensure that you protect yourself from API changes, which will only be made in major revisions.
 
-# Kicking Off a Job
+# Roughly, What Am I Getting Myself Into?
 
-When using the EMR UI, there are several sample jobs that Amazon supplies.  The assets for these sample jobs are hosted on S3 and publicly available meaning you can run this code as-is (supplying your AWS credentials appropriately) and ```JobFlow#run``` will return the ID of the job flow.
+If you're familiar with the AWS EMR UI, you'll recall there are sample jobs Amazon supplies to help us get familiar with EMR.  Here's how you'd kick off the "Cloudburst (Custom Jar)" sample job with Elasticity.  You can run this code as-is (supplying your AWS credentials and an output location) and ```JobFlow#run``` will return the ID of the job flow.
 
 ```ruby
 require 'elasticity'
@@ -31,11 +31,14 @@ require 'elasticity'
 # Create a job flow with your AWS credentials
 jobflow = Elasticity::JobFlow.new('AWS access key', 'AWS secret key')
 
+# Omit credentials to use the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables
+# jobflow = Elasticity::JobFlow.new
+
 # This is the first step in the jobflow - running a custom jar
 step = Elasticity::CustomJarStep.new('s3n://elasticmapreduce/samples/cloudburst/cloudburst.jar')
 
 # Here are the arguments to pass to the jar
-step.arguments = %w(s3n://elasticmapreduce/samples/cloudburst/input/s_suis.br s3n://elasticmapreduce/samples/cloudburst/input/100k.br s3n://slif-output/cloudburst/output/2012-06-22 36 3 0 1 240 48 24 24 128 16)
+step.arguments = %w(s3n://elasticmapreduce/samples/cloudburst/input/s_suis.br s3n://elasticmapreduce/samples/cloudburst/input/100k.br s3n://OUTPUT_BUCKET/cloudburst/output/2012-06-22 36 3 0 1 240 48 24 24 128 16)
 
 # Add the step to the jobflow
 jobflow.add_step(step)
@@ -44,7 +47,7 @@ jobflow.add_step(step)
 jobflow.run
 ```
 
-Note that this example is only for ```CustomJarStep```.  ```PigStep``` and ```HiveStep``` will have different means of passing parameters.
+Note that this example is only for ```CustomJarStep```.  Other steps will have different means of passing parameters.
 
 # Working with Job Flows
 
@@ -54,7 +57,8 @@ Job flows are the center of the EMR universe.  The general order of operations i
   1. Specify options.
   1. (optional) Configure instance groups.
   1. (optional) Add bootstrap actions.
-  1. Create steps.
+  1. Add steps.
+  1. (optional) Upload assets.
   1. Run the job flow.
   1. (optional) Add additional steps.
   1. (optional) Shutdown the job flow.
@@ -64,39 +68,49 @@ Job flows are the center of the EMR universe.  The general order of operations i
 Only your AWS credentials are needed.
 
 ```ruby
+# Manually specify AWS credentials
 jobflow = Elasticity::JobFlow.new('AWS access key', 'AWS secret key')
+
+# Use the standard environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)
+jobflow = Elasticity::JobFlow.new
 ```
 
 If you want to access a job flow that's already running:
 
 ```ruby
+# Manually specify AWS credentials
 jobflow = Elasticity::JobFlow.from_jobflow_id('AWS access key', 'AWS secret key', 'jobflow ID', 'region')
+
+# Use the standard environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)
+jobflow = Elasticity::JobFlow.from_jobflow_id(nil, nil, 'jobflow ID', 'region')
 ```
 
 This is useful if you'd like to attach to a running job flow and add more steps, etc.  The ```region``` parameter is necessary because job flows are only accessible from the the API when you connect to the same endpoint that created them (e.g. us-west-1).  If you don't specify the ```region``` parameter, us-east-1 is assumed.
 
-## 2 - Specifying Job Flow Options
+## 2 - Specifying Options
 
 Configuration job flow options, shown below with default values.  Note that these defaults are subject to change - they are reasonable defaults at the time(s) I work on them (e.g. the latest version of Hadoop).
 
 These options are sent up as part of job flow submission (i.e. ```JobFlow#run```), so be sure to configure these before running the job.
 
 ```ruby
+jobflow.name                              = 'Elasticity Job Flow'
+
 jobflow.action_on_failure                 = 'TERMINATE_JOB_FLOW'
+jobflow.keep_job_flow_alive_when_no_steps = false
 jobflow.ami_version                       = 'latest'
-jobflow.ec2_key_name                      = 'default'
-jobflow.ec2_subnet_id                     = nil
-jobflow.hadoop_version                    = '0.20.205'
-jobflow.keep_job_flow_alive_when_no_steps = true
+jobflow.hadoop_version                    = '1.0.3'
 jobflow.log_uri                           = nil
-jobflow.name                              = 'Elasticity Job Flow'
+
+jobflow.ec2_key_name                      = nil
+jobflow.ec2_subnet_id                     = nil
 jobflow.placement                         = 'us-east-1a'
 jobflow.instance_count                    = 2
 jobflow.master_instance_type              = 'm1.small'
 jobflow.slave_instance_type               = 'm1.small'
 ```
 
-## 3 - Configuring Instance Groups (optional)
+## 3 - Configure Instance Groups (optional)
 
 Technically this is optional since Elasticity creates MASTER and CORE instance groups for you (one m1.small instance in each).  If you'd like your jobs to finish in an appreciable amount of time, you'll want to at least add a few instances to the CORE group :)
 
@@ -142,10 +156,23 @@ ig.set_spot_instances(0.25)         # Makes this a SPOT group with a $0.25 bid p
 jobflow.set_core_instance_group(ig)
 ```
 
-## 4 - Adding Bootstrap Actions (optional)
+## 4 - Add Bootstrap Actions (optional)
 
 Bootstrap actions are run as part of setting up the job flow, so be sure to configure these before running the job.
 
+### Bootstrap Actions
+
+With the basic ```BootstrapAction``` you specify everything about the action - the script, options and arguments.
+
+```ruby
+action = Elasticity::BootstrapAction.new('s3n://my-bucket/my-script', '-g', '100')
+jobflow.add_bootstrap_action(action)
+```
+
+### Hadoop Bootstrap Actions
+
+`HadoopBootstrapAction` handles passing Hadoop configuration options through.
+
 ```ruby
 [
   Elasticity::HadoopBootstrapAction.new('-m', 'mapred.map.tasks=101'),
@@ -156,7 +183,16 @@ Bootstrap actions are run as part of setting up the job flow, so be sure to conf
 end
 ```
 
-## 5 - Adding Steps
+### Hadoop File Bootstrap Actions
+
+With EMR's current limit of 15 bootstrap actions, chances are you're going to create a configuration file full of your options and opt to use that instead of passing all the options individually.  In that case, use the ```HadoopFileBootstrapAction```, supplying the location of your configuration file.
+
+```ruby
+action = Elasticity::HadoopFileBootstrapAction.new('s3n://my-bucket/job-config.xml')
+jobflow.add_bootstrap_action(action)
+```
+
+## 5 - Add Steps
 
 Each type of step has ```#name``` and ```#action_on_failure``` fields that can be overridden.  Apart from that, steps are configured differently - exhaustively described below.
 
@@ -235,7 +271,24 @@ jar_step.arguments = ['arg1', 'arg2']
 jobflow.add_step(jar_step)
 ```
 
-## 6 - Running the Job Flow
+## 6 - Upload Assets (optional)
+
+This isn't part of ```JobFlow```; more of an aside :)  Elasticity provides a very basic means of uploading assets to S3 so that your EMR job has access to them.  For example, a TSV file with a range of valid values, join tables, etc.
+
+```ruby
+# Specify the bucket and AWS credentials
+s3 = Elasticity::SyncToS3('my-bucket', 'access', 'secret')
+
+# Use the standard environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)
+# s3 = Elasticity::SyncToS3('my-bucket')
+
+# Recursively sync the contents of '/some/parent/dir' under the remote location 'remote-dir/this-job/assets'
+s3.sync('/some/parent/dir', 'remote-dir/this-job/assets')
+```
+
+If the files already exist, there is an MD5 checksum check.  If the checksums are the same, the file will be skipped.  Now you can use something like ```s3n://my-bucket/remote-dir/this-job/assets/join.tsv``` in your EMR jobs.
+
+## 7 - Run the Job Flow
 
 Submit the job flow to Amazon, storing the ID of the running job flow.
 
@@ -243,11 +296,11 @@ Submit the job flow to Amazon, storing the ID of the running job flow.
 jobflow_id = jobflow.run
 ```
 
-## 7 - Adding Additional Steps (optional)
+## 8 - Add Additional Steps (optional)
 
 Steps can be added to a running jobflow just by calling ```#add_step``` on the job flow exactly how you add them prior to submitting the job.
 
-## 8 - Shutting Down the Job Flow (optional)
+## 9 - Shut Down the Job Flow (optional)
 
 By default, job flows are set to terminate when there are no more running steps.  You can tell the job flow to stay alive when it has nothing left to do:
 

data/elasticity.gemspec

@@ -13,12 +13,15 @@ Gem::Specification.new do |s|
 
   s.add_dependency('rest-client')
   s.add_dependency('nokogiri')
+  s.add_dependency('fog')
 
   s.add_development_dependency('rake')
   s.add_development_dependency('rspec', '~> 2.11.0')
+  s.add_development_dependency('timecop')
+  s.add_development_dependency('fakefs', '~> 0.4')
 
   s.files         = `git ls-files`.split("\n")
-  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.test_files    = `git ls-files -- {spec,features}/*`.split("\n")
   s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
   s.require_paths = %w(lib)
 end

data/lib/elasticity.rb

@@ -3,13 +3,18 @@ require 'time'
 
 require 'rest_client'
 require 'nokogiri'
+require 'fog'
 
 require 'elasticity/support/conditional_raise'
 
 require 'elasticity/aws_request'
 require 'elasticity/emr'
 
+require 'elasticity/sync_to_s3'
+
+require 'elasticity/bootstrap_action'
 require 'elasticity/hadoop_bootstrap_action'
+require 'elasticity/hadoop_file_bootstrap_action'
 require 'elasticity/job_flow_step'
 
 require 'elasticity/job_flow'

data/lib/elasticity/aws_request.rb

@@ -1,5 +1,8 @@
 module Elasticity
 
+  class MissingKeyError < StandardError;
+  end
+
   class AwsRequest
 
     attr_reader :access_key
@@ -10,9 +13,9 @@ module Elasticity
     # Supported values for options:
     #  :region - AWS region (e.g. us-west-1)
     #  :secure - true or false, default true.
-    def initialize(access, secret, options = {})
-      @access_key = access
-      @secret_key = secret
+    def initialize(access=nil, secret=nil, options={})
+      @access_key = get_access_key(access)
+      @secret_key = get_secret_key(secret)
       @host = "elasticmapreduce.#{{:region => 'us-east-1'}.merge(options)[:region]}.amazonaws.com"
       @protocol = {:secure => true}.merge(options)[:secure] ? 'https' : 'http'
     end
@@ -38,20 +41,33 @@ module Elasticity
 
     private
 
+    def get_access_key(access)
+      return access if access
+      return ENV['AWS_ACCESS_KEY_ID'] if ENV['AWS_ACCESS_KEY_ID']
+      raise MissingKeyError, 'Please provide an access key or set AWS_ACCESS_KEY_ID.'
+    end
+
+    def get_secret_key(secret)
+      return secret if secret
+      return ENV['AWS_SECRET_ACCESS_KEY'] if ENV['AWS_SECRET_ACCESS_KEY']
+      raise MissingKeyError, 'Please provide a secret key or set AWS_ACCESS_KEY_ID.'
+    end
+
     # (Used from RightScale's right_aws gem.)
     # EC2, SQS, SDB and EMR requests must be signed by this guy.
     # See: http://docs.amazonwebservices.com/AmazonSimpleDB/2007-11-07/DeveloperGuide/index.html?REST_RESTAuth.html
     #      http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1928
     def sign_params(service_hash)
-      uri = '/' # TODO: Why are we hard-coding this?
-      service_hash["AWSAccessKeyId"] = @access_key
-      service_hash["Timestamp"] = Time.now.utc.strftime("%Y-%m-%dT%H:%M:%S.000Z")
-      service_hash["SignatureVersion"] = "2"
-      service_hash["SignatureMethod"] = "HmacSHA256"
+      service_hash.merge!({
+        'AWSAccessKeyId' => @access_key,
+        'Timestamp' => Time.now.utc.strftime('%Y-%m-%dT%H:%M:%S.000Z'),
+        'SignatureVersion' => '2',
+        'SignatureMethod' => 'HmacSHA256'
+      })
       canonical_string = service_hash.keys.sort.map do |key|
         "#{AwsRequest.aws_escape(key)}=#{AwsRequest.aws_escape(service_hash[key])}"
       end.join('&')
-      string_to_sign = "POST\n#{@host.downcase}\n#{uri}\n#{canonical_string}"
+      string_to_sign = "POST\n#{@host.downcase}\n/\n#{canonical_string}"
       signature = AwsRequest.aws_escape(Base64.encode64(OpenSSL::HMAC.digest("sha256", @secret_key, string_to_sign)).strip)
       "#{canonical_string}&Signature=#{signature}"
     end
@@ -95,12 +111,8 @@ module Elasticity
     end
 
     # (Used from Rails' ActiveSupport)
-    def self.camelize(lower_case_and_underscored_word, first_letter_in_uppercase = true)
-      if first_letter_in_uppercase
-        lower_case_and_underscored_word.to_s.gsub(/\/(.?)/) { "::" + $1.upcase }.gsub(/(^|_)(.)/) { $2.upcase }
-      else
-        lower_case_and_underscored_word.first + camelize(lower_case_and_underscored_word)[1..-1]
-      end
+    def self.camelize(word)
+      word.to_s.gsub(/\/(.?)/) { "::" + $1.upcase }.gsub(/(^|_)(.)/) { $2.upcase }
     end
 
     # AWS error responses all follow the same form.  Extract the message from

data/lib/elasticity/bootstrap_action.rb

@@ -0,0 +1,29 @@
+module Elasticity
+
+  class BootstrapAction
+
+    attr_accessor :name
+    attr_accessor :option
+    attr_accessor :value
+    attr_accessor :script
+
+    def initialize(script, option, value)
+      @name = 'Elasticity Bootstrap Action'
+      @option = option
+      @value = value
+      @script = script
+    end
+
+    def to_aws_bootstrap_action
+      {
+        :name => @name,
+        :script_bootstrap_action => {
+          :path => @script,
+          :args => [@option, @value]
+        }
+      }
+    end
+
+  end
+
+end

data/lib/elasticity/hadoop_bootstrap_action.rb

@@ -1,25 +1,12 @@
 module Elasticity
 
-  class HadoopBootstrapAction
-
-    attr_accessor :name
-    attr_accessor :option
-    attr_accessor :value
+  class HadoopBootstrapAction < BootstrapAction
 
     def initialize(option, value)
       @name = 'Elasticity Bootstrap Action (Configure Hadoop)'
       @option = option
       @value = value
-    end
-
-    def to_aws_bootstrap_action
-      {
-        :name => @name,
-        :script_bootstrap_action => {
-          :path => 's3n://elasticmapreduce/bootstrap-actions/configure-hadoop',
-          :args => [@option, @value]
-        }
-      }
+      @script = 's3n://elasticmapreduce/bootstrap-actions/configure-hadoop'
     end
 
   end

data/lib/elasticity/hadoop_file_bootstrap_action.rb

@@ -0,0 +1,14 @@
+module Elasticity
+
+  class HadoopFileBootstrapAction < BootstrapAction
+
+    def initialize(config_file)
+      @name = 'Elasticity Bootstrap Action (Configure Hadoop via File)'
+      @option = '--mapred-config-file'
+      @value = config_file
+      @script = 's3n://elasticmapreduce/bootstrap-actions/configure-hadoop'
+    end
+
+  end
+
+end

data/lib/elasticity/job_flow.rb

@@ -19,7 +19,10 @@ module Elasticity
     attr_accessor :ec2_subnet_id
     attr_accessor :placement
 
-    def initialize(access, secret)
+    attr_reader :access_key
+    attr_reader :secret_key
+    
+    def initialize(access=nil, secret=nil)
       @action_on_failure = 'TERMINATE_JOB_FLOW'
       @hadoop_version = '1.0.3'
       @name = 'Elasticity Job Flow'
@@ -27,8 +30,8 @@ module Elasticity
       @keep_job_flow_alive_when_no_steps = false
       @placement = 'us-east-1a'
 
-      @access = access
-      @secret = secret
+      @access_key = access
+      @secret_key = secret
 
       @bootstrap_actions = []
       @jobflow_steps = []
@@ -41,8 +44,8 @@ module Elasticity
       @master_instance_type = 'm1.small'
       @slave_instance_type = 'm1.small'
 
-      @access = access
-      @secret = secret
+      @access_key = access
+      @secret_key = secret
     end
 
     def self.from_jobflow_id(access, secret, jobflow_id, region = 'us-east-1')
@@ -122,7 +125,7 @@ module Elasticity
 
     def emr
       @region ||= @placement.match(/(\w+-\w+-\d+)/)[0]
-      @emr ||= Elasticity::EMR.new(@access, @secret, :region => @region)
+      @emr ||= Elasticity::EMR.new(@access_key, @secret_key, :region => @region)
     end
 
     def is_jobflow_running?

data/lib/elasticity/sync_to_s3.rb

@@ -0,0 +1,76 @@
+module Elasticity
+
+  class NoBucketError < StandardError; end
+  class NoDirectoryError < StandardError; end
+
+  class SyncToS3
+
+    attr_reader :access_key
+    attr_reader :secret_key
+    attr_reader :bucket_name
+
+    def initialize(bucket, access=nil, secret=nil)
+      @access_key = get_access_key(access)
+      @secret_key = get_secret_key(secret)
+      @bucket_name = bucket
+    end
+
+    def sync(local, remote)
+      raise_unless bucket, NoBucketError, "Bucket '#@bucket_name' does not exist"
+      raise_unless File.directory?(local), NoDirectoryError, "Directory '#{local}' does not exist or is not a directory"
+      sync_dir(local, remote)
+    end
+
+    private
+
+    def sync_dir(local, remote)
+      Dir.glob(File.join([local, '*'])).each do |entry|
+        if File.directory?(entry)
+          sync_dir(entry, [remote, File.basename(entry)].join('/'))
+        else
+          sync_file(entry, remote)
+        end
+      end
+    end
+
+    def sync_file(file_name, remote_dir)
+      remote_dir = remote_dir.gsub(/^(\/)/, '')
+      remote_path = (remote_dir.empty?) ? (File.basename(file_name)) : [remote_dir, File.basename(file_name)].join('/')
+      metadata = bucket.files.head(remote_path)
+      return if metadata && metadata.etag == Digest::MD5.file(file_name).to_s
+
+      bucket.files.create({
+        :key => remote_path,
+        :body => File.open(file_name),
+        :public => false
+      })
+    end
+
+    def bucket
+      index = s3.directories.index { |d| d.key == @bucket_name }
+      @bucket ||= index ? s3.directories[index] : nil
+    end
+
+    def s3
+      @connection ||= Fog::Storage.new({
+        :provider => 'AWS',
+        :aws_access_key_id => @access_key,
+        :aws_secret_access_key => @secret_key
+      })
+    end
+
+    def get_access_key(access)
+      return access if access
+      return ENV['AWS_ACCESS_KEY_ID'] if ENV['AWS_ACCESS_KEY_ID']
+      raise MissingKeyError, 'Please provide an access key or set AWS_ACCESS_KEY_ID.'
+    end
+
+    def get_secret_key(secret)
+      return secret if secret
+      return ENV['AWS_SECRET_ACCESS_KEY'] if ENV['AWS_SECRET_ACCESS_KEY']
+      raise MissingKeyError, 'Please provide a secret key or set AWS_SECRET_ACCESS_KEY.'
+    end
+
+  end
+
+end

data/lib/elasticity/version.rb

@@ -1,3 +1,3 @@
 module Elasticity
-  VERSION = '2.3.1'
+  VERSION = '2.4'
 end

data/spec/lib/elasticity/aws_request_spec.rb

@@ -1,15 +1,81 @@
 describe Elasticity::AwsRequest do
 
   before do
-    Time.stub(:now).and_return(Time.at(1302461096))
+    Timecop.freeze(Time.at(1302461096))
   end
 
   subject do
     Elasticity::AwsRequest.new('access', 'secret')
   end
 
-  its(:access_key) { should == 'access' }
-  its(:secret_key) { should == 'secret' }
+  describe '#initialize' do
+
+    context 'when access and/or secret keys are provided' do
+      it 'should set them to the provided values' do
+        subject.access_key.should == 'access'
+        subject.secret_key.should == 'secret'
+      end
+    end
+
+    context 'when either access or secret key is not provided or nil' do
+
+      context 'when the proper environment variables are set' do
+
+        context 'when access and secret key are not provided' do
+          let(:default_values) { Elasticity::AwsRequest.new }
+          before do
+            ENV.stub(:[]).with('AWS_ACCESS_KEY_ID').and_return('ENV_ACCESS')
+            ENV.stub(:[]).with('AWS_SECRET_ACCESS_KEY').and_return('ENV_SECRET')
+          end
+          it 'should set access and secret keys' do
+            default_values.access_key.should == 'ENV_ACCESS'
+            default_values.secret_key.should == 'ENV_SECRET'
+          end
+        end
+
+        context 'when access and secret key are nil' do
+          let(:nil_values) { Elasticity::AwsRequest.new(nil, nil) }
+          before do
+            ENV.stub(:[]).with('AWS_ACCESS_KEY_ID').and_return('ENV_ACCESS')
+            ENV.stub(:[]).with('AWS_SECRET_ACCESS_KEY').and_return('ENV_SECRET')
+          end
+          it 'should set access and secret keys' do
+            nil_values.access_key.should == 'ENV_ACCESS'
+            nil_values.secret_key.should == 'ENV_SECRET'
+          end
+        end
+
+      end
+
+      context 'when the environment variables are not set' do
+        let(:missing_something) { Elasticity::AwsRequest.new }
+        context 'when the access key is not set' do
+          before do
+            ENV.stub(:[]).with('AWS_ACCESS_KEY_ID').and_return(nil)
+            ENV.stub(:[]).with('AWS_SECRET_ACCESS_KEY').and_return('_')
+          end
+          it 'should raise an error' do
+            expect {
+              missing_something.access_key
+            }.to raise_error(Elasticity::MissingKeyError, 'Please provide an access key or set AWS_ACCESS_KEY_ID.')
+          end
+        end
+        context 'when the secret key is not set' do
+          before do
+            ENV.stub(:[]).with('AWS_ACCESS_KEY_ID').and_return('_')
+            ENV.stub(:[]).with('AWS_SECRET_ACCESS_KEY').and_return(nil)
+          end
+          it 'should raise an error' do
+            expect {
+              missing_something.access_key
+            }.to raise_error(Elasticity::MissingKeyError, 'Please provide a secret key or set AWS_ACCESS_KEY_ID.')
+          end
+        end
+      end
+
+    end
+
+  end
 
   describe '#host' do
 

data/spec/lib/elasticity/bootstrap_action_spec.rb

@@ -0,0 +1,25 @@
+describe Elasticity::BootstrapAction do
+
+  subject do
+    Elasticity::BootstrapAction.new('script', 'option', 'value')
+  end
+
+  its(:name) { should == 'Elasticity Bootstrap Action' }
+  its(:option) { should == 'option' }
+  its(:value) { should == 'value' }
+  its(:script) { should == 'script' }
+
+  describe '#to_aws_bootstrap_action' do
+    it 'should create a bootstrap action' do
+      subject.to_aws_bootstrap_action.should ==
+        {
+          :name => 'Elasticity Bootstrap Action',
+          :script_bootstrap_action => {
+            :path => 'script',
+            :args => %w(option value)
+          }
+        }
+    end
+  end
+
+end

data/spec/lib/elasticity/hadoop_bootstrap_action_spec.rb

@@ -4,23 +4,11 @@ describe Elasticity::HadoopBootstrapAction do
     Elasticity::HadoopBootstrapAction.new('option', 'value')
   end
 
+  it { should be_a Elasticity::BootstrapAction }
+
   its(:name) { should == 'Elasticity Bootstrap Action (Configure Hadoop)' }
   its(:option) { should == 'option' }
   its(:value) { should == 'value' }
-
-  describe '#to_aws_bootstrap_action' do
-
-    it 'should create a bootstrap action' do
-      subject.to_aws_bootstrap_action.should ==
-        {
-          :name => 'Elasticity Bootstrap Action (Configure Hadoop)',
-          :script_bootstrap_action => {
-            :path => 's3n://elasticmapreduce/bootstrap-actions/configure-hadoop',
-            :args => ['option', 'value']
-          }
-        }
-    end
-
-  end
+  its(:script) { should == 's3n://elasticmapreduce/bootstrap-actions/configure-hadoop' }
 
 end

data/spec/lib/elasticity/hadoop_file_bootstrap_action_spec.rb

@@ -0,0 +1,14 @@
+describe Elasticity::HadoopFileBootstrapAction do
+
+  subject do
+    Elasticity::HadoopFileBootstrapAction.new('config_file')
+  end
+
+  it { should be_a Elasticity::BootstrapAction }
+
+  its(:name) { should == 'Elasticity Bootstrap Action (Configure Hadoop via File)' }
+  its(:option) { should == '--mapred-config-file' }
+  its(:value) { should == 'config_file' }
+  its(:script) { should == 's3n://elasticmapreduce/bootstrap-actions/configure-hadoop' }
+
+end

data/spec/lib/elasticity/job_flow_spec.rb

@@ -4,6 +4,8 @@ describe Elasticity::JobFlow do
     Elasticity::JobFlow.new('access', 'secret')
   end
 
+  its(:access_key) { should == 'access' }
+  its(:secret_key) { should == 'secret' }
   its(:action_on_failure) { should == 'TERMINATE_JOB_FLOW' }
   its(:ec2_key_name) { should == nil }
   its(:ec2_subnet_id) { should == nil }
@@ -17,6 +19,18 @@ describe Elasticity::JobFlow do
   its(:keep_job_flow_alive_when_no_steps) { should == false }
   its(:placement) { should == 'us-east-1a' }
 
+  describe '.initialize' do
+    it 'should set the access and secret keys to nil by default' do
+      Elasticity::JobFlow.new.tap do |j|
+        j.access_key.should == nil
+        j.secret_key.should == nil
+      end
+      Elasticity::JobFlow.new('_') do |j|
+        j.secret_key.should == nil
+      end
+    end
+  end
+
   describe '#instance_count=' do
 
     context 'when set to more than 1' do

data/spec/lib/elasticity/sync_to_s3_spec.rb

@@ -0,0 +1,240 @@
+describe Elasticity::SyncToS3 do
+
+  include FakeFS::SpecHelpers
+  Fog.mock!
+
+  let(:bucket_name) { 'TEST_BUCKET' }
+  let(:sync_to_s3) { Elasticity::SyncToS3.new(bucket_name, '_', '_') }
+  let(:s3) { Fog::Storage.new({:provider => 'AWS', :aws_access_key_id => '', :aws_secret_access_key => ''}) }
+
+  before do
+    Fog::Mock.reset
+    sync_to_s3.stub(:s3).and_return(s3)
+  end
+
+  describe '#initialize' do
+
+    describe 'basic assignment' do
+
+      it 'should set the proper values' do
+        sync = Elasticity::SyncToS3.new('bucket', 'access', 'secret')
+        sync.access_key.should == 'access'
+        sync.secret_key.should == 'secret'
+        sync.bucket_name.should == 'bucket'
+      end
+
+    end
+
+    context 'when access and secret keys are nil' do
+
+      let(:both_keys_nil) { Elasticity::SyncToS3.new('_', nil, nil) }
+      let(:both_keys_missing) { Elasticity::SyncToS3.new('_') }
+
+      before do
+        ENV.stub(:[]).with('AWS_ACCESS_KEY_ID').and_return(access_key)
+        ENV.stub(:[]).with('AWS_SECRET_ACCESS_KEY').and_return(secret_key)
+      end
+
+      context 'when environment variables are present' do
+        let(:access_key) { 'ENV_ACCESS' }
+        let(:secret_key) { 'ENV_SECRET' }
+        it 'should assign them to the keys' do
+          both_keys_nil.access_key.should == 'ENV_ACCESS'
+          both_keys_nil.secret_key.should == 'ENV_SECRET'
+
+          both_keys_missing.access_key.should == 'ENV_ACCESS'
+          both_keys_missing.secret_key.should == 'ENV_SECRET'
+        end
+      end
+
+      context 'when environment variables are not present' do
+
+        context 'when access is not set' do
+          let(:access_key) { nil }
+          let(:secret_key) { '_' }
+          it 'should raise an error' do
+            expect {
+              both_keys_nil # Trigger instantiation
+            }.to raise_error(Elasticity::MissingKeyError, 'Please provide an access key or set AWS_ACCESS_KEY_ID.')
+          end
+        end
+
+        context 'when secret is not set' do
+          let(:access_key) { '_' }
+          let(:secret_key) { nil }
+          it 'should raise an error' do
+            expect {
+              both_keys_nil # Trigger instantiation
+            }.to raise_error(Elasticity::MissingKeyError, 'Please provide a secret key or set AWS_SECRET_ACCESS_KEY.')
+          end
+        end
+
+      end
+
+    end
+
+  end
+
+  describe '#sync' do
+
+    context 'when the bucket exists' do
+
+      before do
+        s3.directories.create(:key => bucket_name)
+      end
+
+      context 'when the local directory exists' do
+        before do
+          FileUtils.mkdir('GOOD_DIR')
+        end
+        it 'should sync that directory' do
+          sync_to_s3.should_receive(:sync_dir).with('GOOD_DIR', 'REMOTE_DIR')
+          sync_to_s3.sync('GOOD_DIR', 'REMOTE_DIR')
+        end
+      end
+
+      context 'when the local directory does not exist' do
+        it 'should raise an error' do
+          expect {
+            sync_to_s3.sync('BAD_DIR', '_')
+          }.to raise_error(Elasticity::NoDirectoryError, "Directory 'BAD_DIR' does not exist or is not a directory")
+        end
+      end
+
+      context 'when the local directory is not a directory' do
+        before do
+          FileUtils.touch('NOT_A_DIR')
+        end
+        it 'should raise an error' do
+          expect {
+            sync_to_s3.sync('NOT_A_DIR', '_')
+          }.to raise_error(Elasticity::NoDirectoryError, "Directory 'NOT_A_DIR' does not exist or is not a directory")
+        end
+      end
+
+    end
+
+    context 'when the bucket does not exist' do
+      let(:bucket_name) { 'BAD_BUCKET' }
+      it 'should raise an error' do
+        expect {
+          sync_to_s3.sync('_', '_')
+        }.to raise_error(Elasticity::NoBucketError, "Bucket 'BAD_BUCKET' does not exist")
+      end
+    end
+
+  end
+
+  describe '#sync_dir' do
+
+    before do
+      s3.directories.create(:key => bucket_name)
+
+      FileUtils.makedirs(File.join(%w(local_dir sub_dir_1)))
+      FileUtils.makedirs(File.join(%w(local_dir sub_dir_2)))
+
+      FileUtils.touch(File.join(%w(local_dir file_1)))
+      FileUtils.touch(File.join(%w(local_dir file_2)))
+      FileUtils.touch(File.join(%w(local_dir sub_dir_1 file_3)))
+      FileUtils.touch(File.join(%w(local_dir sub_dir_1 file_4)))
+      FileUtils.touch(File.join(%w(local_dir sub_dir_2 file_5)))
+      FileUtils.touch(File.join(%w(local_dir sub_dir_2 file_6)))
+    end
+
+    it 'should recursively sync all files and directories' do
+      sync_to_s3.send(:sync_dir, 'local_dir', 'remote_dir')
+
+      %w(
+        remote_dir/file_1
+        remote_dir/file_2
+        remote_dir/sub_dir_1/file_3
+        remote_dir/sub_dir_1/file_4
+        remote_dir/sub_dir_2/file_5
+        remote_dir/sub_dir_2/file_6
+      ).each do |key|
+        s3.directories[0].files.map(&:key).should include(key)
+      end
+    end
+
+  end
+
+  describe '#sync_file' do
+
+    let(:local_dir) { '/tmp' }
+    let(:file_name) { 'test.out' }
+    let(:full_path) { File.join([local_dir, file_name]) }
+    let(:remote_dir) { 'job/assets' }
+    let(:remote_path) { "#{remote_dir}/#{file_name}"}
+    let(:file_data) { 'Some test content' }
+
+    before do
+      s3.directories.create(:key => bucket_name)
+      FileUtils.makedirs(local_dir)
+      File.open(full_path, 'w') {|f| f.write(file_data) }
+    end
+
+    it 'should write the specified file into the remote directory' do
+      sync_to_s3.send(:sync_file, full_path, remote_dir)
+      s3.directories[0].files.head(remote_path).should_not be_nil
+    end
+
+    it 'should write the contents of the file' do
+      sync_to_s3.send(:sync_file, full_path, remote_dir)
+      s3.directories[0].files.head(remote_path).body.should == file_data
+    end
+
+    it 'should write the remote file without public access' do
+      sync_to_s3.send(:sync_file, full_path, remote_dir)
+      s3.directories[0].files.head(remote_path).public_url.should be_nil
+    end
+
+    it 'should not write identical content' do
+      sync_to_s3.send(:sync_file, full_path, remote_dir)
+      last_modified = s3.directories[0].files.head(remote_path).last_modified
+      Timecop.travel(Time.now + 60)
+      sync_to_s3.send(:sync_file, full_path, remote_dir)
+      s3.directories[0].files.head(remote_path).last_modified.should == last_modified
+    end
+
+    context 'when remote dir is a corner case value' do
+      before do
+        sync_to_s3.send(:sync_file, full_path, remote_dir)
+      end
+
+      context 'when remote dir is empty' do
+        let(:remote_dir) {''}
+        it 'should place files in the root without a bunk empty folder name' do
+          s3.directories[0].files.head(file_name).should_not be_nil
+        end
+      end
+
+      context 'when remote dir is /' do
+        let(:remote_dir) {'/'}
+        it 'should place files in the root without a bunk empty folder name' do
+          s3.directories[0].files.head(file_name).should_not be_nil
+        end
+      end
+
+      context 'when remote dir starts with a /' do
+        let(:remote_dir) {'/starts_with_slash'}
+        it 'should place files in the root without a bunk empty folder name' do
+          s3.directories[0].files.head('starts_with_slash/test.out').should_not be_nil
+        end
+      end
+    end
+
+  end
+
+  describe '#s3' do
+    let(:connection_test) { Elasticity::SyncToS3.new('_', 'access', 'secret') }
+    it 'should connect to S3 using the specified credentials' do
+      Fog::Storage.should_receive(:new).with({
+        :provider => 'AWS',
+        :aws_access_key_id => 'access',
+        :aws_secret_access_key => 'secret'
+      }).and_return('GOOD_CONNECTION')
+      connection_test.send(:s3).should == 'GOOD_CONNECTION'
+    end
+  end
+
+end

data/spec/spec_helper.rb

@@ -2,6 +2,9 @@ require 'rubygems'
 require 'bundler/setup'
 require 'elasticity'
 
-ENV["RAILS_ENV"] ||= 'test'
+require 'timecop'
+require 'fakefs/spec_helpers'
 
-Dir[File.join(File.dirname(__FILE__), "support", "**", "*.rb")].each { |f| require f }
+ENV['RAILS_ENV'] ||= 'test'
+
+Dir[File.join(File.dirname(__FILE__), 'support', '**', '*.rb')].each { |f| require f }

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: elasticity
 version: !ruby/object:Gem::Version
-  version: 2.3.1
+  version: '2.4'
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-08-23 00:00:00.000000000 Z
+date: 2012-09-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rest-client
@@ -43,6 +43,22 @@ dependencies:
     - - ! '>='
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: fog
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
@@ -75,6 +91,38 @@ dependencies:
     - - ~>
       - !ruby/object:Gem::Version
         version: 2.11.0
+- !ruby/object:Gem::Dependency
+  name: timecop
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: fakefs
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '0.4'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '0.4'
 description: Streamlined, Programmatic access to Amazon's Elastic Map Reduce service,
   driven by the Sharethrough team's requirements for belting out EMR jobs.
 email: 
@@ -94,9 +142,11 @@ files:
 - elasticity.gemspec
 - lib/elasticity.rb
 - lib/elasticity/aws_request.rb
+- lib/elasticity/bootstrap_action.rb
 - lib/elasticity/custom_jar_step.rb
 - lib/elasticity/emr.rb
 - lib/elasticity/hadoop_bootstrap_action.rb
+- lib/elasticity/hadoop_file_bootstrap_action.rb
 - lib/elasticity/hive_step.rb
 - lib/elasticity/instance_group.rb
 - lib/elasticity/job_flow.rb
@@ -106,11 +156,14 @@ files:
 - lib/elasticity/pig_step.rb
 - lib/elasticity/streaming_step.rb
 - lib/elasticity/support/conditional_raise.rb
+- lib/elasticity/sync_to_s3.rb
 - lib/elasticity/version.rb
 - spec/lib/elasticity/aws_request_spec.rb
+- spec/lib/elasticity/bootstrap_action_spec.rb
 - spec/lib/elasticity/custom_jar_step_spec.rb
 - spec/lib/elasticity/emr_spec.rb
 - spec/lib/elasticity/hadoop_bootstrap_action_spec.rb
+- spec/lib/elasticity/hadoop_file_bootstrap_action_spec.rb
 - spec/lib/elasticity/hive_step_spec.rb
 - spec/lib/elasticity/instance_group_spec.rb
 - spec/lib/elasticity/job_flow_integration_spec.rb
@@ -121,6 +174,7 @@ files:
 - spec/lib/elasticity/pig_step_spec.rb
 - spec/lib/elasticity/streaming_step_spec.rb
 - spec/lib/elasticity/support/conditional_raise_spec.rb
+- spec/lib/elasticity/sync_to_s3_spec.rb
 - spec/spec_helper.rb
 - spec/support/be_a_hash_including_matcher.rb
 homepage: http://www.github.com/rslifka/elasticity
@@ -137,7 +191,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 812718050856923137
+      hash: -2552375759855937320
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -146,7 +200,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 812718050856923137
+      hash: -2552375759855937320
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.24
@@ -155,9 +209,11 @@ specification_version: 3
 summary: Streamlined, programmatic access to Amazon's Elastic Map Reduce service.
 test_files:
 - spec/lib/elasticity/aws_request_spec.rb
+- spec/lib/elasticity/bootstrap_action_spec.rb
 - spec/lib/elasticity/custom_jar_step_spec.rb
 - spec/lib/elasticity/emr_spec.rb
 - spec/lib/elasticity/hadoop_bootstrap_action_spec.rb
+- spec/lib/elasticity/hadoop_file_bootstrap_action_spec.rb
 - spec/lib/elasticity/hive_step_spec.rb
 - spec/lib/elasticity/instance_group_spec.rb
 - spec/lib/elasticity/job_flow_integration_spec.rb
@@ -168,5 +224,6 @@ test_files:
 - spec/lib/elasticity/pig_step_spec.rb
 - spec/lib/elasticity/streaming_step_spec.rb
 - spec/lib/elasticity/support/conditional_raise_spec.rb
+- spec/lib/elasticity/sync_to_s3_spec.rb
 - spec/spec_helper.rb
 - spec/support/be_a_hash_including_matcher.rb